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ABSTRACT 


MCCLERNON,  CHRISTOPHER  KYLE.  Human  performance  effects  of  adaptive 
automation  of  various  air  traffic  control  information  processing  fimctions.  (Under  the 
direction  of  Dr.  David  B.  Kaber.) 

Advanced  forms  of  automation  are  being  considered  for  application  to  Air  Traffic 
Control  (ATC)  in  order  to  reduce  controller  workload  and  make  higher  traffic  volumes  more 
manageable.  At  this  point  in  time  there  is  only  limited  knowledge  of  the  implications  of,  for 
example.  Adaptive  Automation  (AA)  on  controller  performance,  workload,  and  Situation 
Awareness  (SA).  The  purpose  of  this  research  was  to  (1)  define  a  measure  of  SA  in  an  ATC 
simulation  that  is  a  sensitive  and  reliable  indicator  of  automation  state  changes  as  part  of  AA; 
(2)  empirically  assess  the  SA  measure  for  use  in  investigating  AA  of  various  ATC 
information  processing  functions;  and  (3)  determine  the  relationship(s)  between  AA  in  the 
ATC  simulation  and  operator  SA.  The  situation  awareness  global  assessment  technique 
(S  AGAT)  was  considered  to  be  an  appropriate  candidate  measure  of  SA  during  AA  of  an 
ATC  simulation. 

An  experiment  designed  to  empirically  assess  the  sensitivity  of  a  SAGAT-based 
approach  included  an  ATC  simulation.  Multitask©,  which  is  capable  of  simulating  five  forms 
of  ATC  information  processing  automation:  manual  control  (no  automation),  information 
acquisition,  information  analysis,  decision  making,  and  action  implementation.  A  secondary, 
signal  detection-based  gauge-monitoring  task  was  used  to  measure  Multitask©  workload  and 
trigger  dynamic  function  allocations  (DFA)  between  manual  and  automated  control.  Eight 
subjects  were  recruited  for  the  study  and  each  performed  under  the  five  modes  of  automation 
twice  (one  repetition).  Performance  measures  were  collected  for  both  the  Multitask© 


simulation  (aircraft  cleared,  conflicts,  and  collisions)  and  the  gauge-monitoring  task 
(workload). 

The  S AGAT  involved  freezing  the  simvilation  three  times  per  trial  and  querying  the 
subjects  on  their  perception,  comprehension,  and  projection  (Level  1, 2,  and  3  SA)  of 
simulation  states.  Subject  responses  were  scored  based  on  “ground  truth”  observations 
recorded  during  stops.  Data  was  analyzed,  first,  on  a  per  stop-basis  in  order  to  identify  any 
effect  of  the  general  type  of  control  (manual  or  automated),  and  then  on  a  per  trial-basis  to 
identify  any  mode  of  automation  effects. 

Results  revealed  that  AA  of  the  action  implementation  aspect  of  ATC  information 
processing  produced  superior  performance  during  the  automated  control  periods  as  part  of 
the  simulation.  In  addition,  automation  of  the  information  analysis  aspect  of  ATC 
information  processing  appeared  to  have  a  positive  effect  on  performance  during  manual 
control  periods  as  part  of  adaptive  conditions.  Subjects  were  able  to  benefit  from  the 
automation  even  during  periods  of  manual  control  of  the  simulation.  Coimter  to 
expectations,  no  significant  findings  were  revealed  for  secondary  task  performance 
(workload).  Situation  awareness  results  revealed  that  subjects  were  better  able  to  project  the 
future  simulation  status  (Level  3  SA)  during  manual  control  periods  of  the  simulation  than 
when  operating  under  automation.  The  S AGAT  data  analysis  revealed  no  significant  effect 
of  the  specific  mode  of  automation  on  SA.  In  general,  these  results  suggest  that  SAGAT  was 
not  a  sensitive  measure  of  SA  in  the  simulated  ATC  task.  This  was  primarily  attributed  to  a 
lack  of  consideration  of  the  relevance  of  particular  aircraft  to  a  controller,  at  any  given  point 
in  time  in  the  simulation,  as  part  of  measurement  technique.  As  a  result,  the  final  objective 
of  determining  relationship(s)  between  AA  in  ATC  and  controller  SA  was  not  achieved. 


The  performance  results  of  this  study  were  in  agreement  with  the  results  of  prior 
research  demonstrating  ATC  information  processing  to  benefit  most  from  lower-order 
automation,  like  action  (clearance)  automation.  The  study  also  further  demonstrated  the  use 
of  a  secondary-task  measure  of  workload  as  a  basis  for  facilitating  AA  in  a  complex  task. 
With  respect  to  SA,  the  study  exposed  a  need  for  future  research  to  assess  the  sensitivity  and 
reliability  of  SAGAT  in  an  ATC  environment  and  the  potential  for  development  of  a  more 
sensitive  SAGAT-based  measure  for  this  domain  that  considers  issues  of  aircraft  relevance 


and  concurrent  controller  goal  sets. 
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1  Introduction 


Air  traffic  control  (ATC)  is  one  of  the  most  complex  and  stressful  jobs  in  the  world. 
Controllers  are  responsible  for  communicating  with  numerous  aircraft  simultaneously,  while 
constructing  complex  three-dimensional  mental  “pictures”  of  the  airspace  environment  in 
their  minds  and  maintaining  strictly  regulated  aircraft  separations  and  procedures.  The 
consequences  for  mistakes  include  costly  delays  and  possible  injuries  or  death;  consequently, 
the  margin  for  error  is  minimal.  Beyond  this,  airspace  is  becoming  increasingly  congested 
(Hopkin,  1999).  One  method  currently  employed  to  alleviate  task  burdens  placed  on  air 
traffic  controllers  is  the  introduction  of  computer  control,  or  automation,  in  aircraft 
information  processing. 

There  are  currently  many  different  forms  of  automation  incorporated  in  ATC,  and 
they  can  have  many  advantages  for  air  traffic  controllers  including  a  reduction  in  workload 
(Laois  &  Giannacourou,  1995),  an  increase  in  system  reliability  (National  Research  Council 
(NRC),  1998),  and  the  capability  to  perform  complex  computations  and  data  management 
(Wickens  &  Hollands,  2000).  However,  automation  in  ATC  can  also  present  many 
disadvantages  (Dillingham,  1998),  including  the  removal  of  the  human  from  the  control  loop 
(Wickens,  1992;  Parasuraman,  Sheridan,  &  Wickens,  2000;  Sanders  &  McCormick,  1993; 
Endsley,  1996;  Endsley  &  Kaber,  1999),  vigilance  decrements  (NRC,  1997),  and 
complacency  (Parasuraman  &  Riley,  1997;  Endsley,  1993;  Laois  &  Giannacourou,  1995; 
Parasuraman  et  al.,  2000). 

The  implementation  of  automation  in  ATC  may  also  compromise  a  controller’s 
ability  to  maintain  situation  awareness  (SA);  or  the  perception  of  elements  in  the 


environment,  the  comprehension  of  their  meaning,  and  the  projection  of  their  ftiture  status 
(Endsley,  1995a).  Situation  awareness  has  been  found  to  be  a  critical  factor  in  successful 
ATC,  and  must  be  addressed  when  designing  automated  systems  for  ATC  (Endsley  &  Jones, 
1995;  Endsley  &  Rodgers,  1994).  New,  advanced  forms  of  automation  are  being  considered 
for  ATC  to  alleviate  die  disadvantages  of  conventional  automation,  and  to  preserve  operator 
SA.  These  advanced  forms  of  automation  involve  providing  computer  assistance  only  when 
assistance  is  needed  by  a  controller,  or  adaptively  applying  automation  to  ATC  information 
processing  functions. 

The  following  sections  define  SA,  discuss  the  importance  of  SA  in  ATC,  describe  the 
role  of  automated  systems  in  ATC,  and  address  the  impact  automation  can  have  on  human 
performance,  including  S  A  in  an  ATC  environment. 

1.1  Situation  Awareness  in  Air  Traffic  Control 

In  March,  1994,  a  United  States  Air  Force  (USAF)  F-16  approached  Pope  Air  Force 
Base  (AFB),  North  Carolina  (NC)  with  ATC  clearance,  attempting  a  straight-in  simulated 
flame-out  (SI-SFO)  approach  -  a  high-speed  final  approach  with  a  very  high  approach  angle. 
The  air  traffic  controller  did  not  notify  the  F-16  of  a  C-130  in  the  pattern,  because  he 
intended  to  simply  sequence  the  C-130  behind  the  F-16.  However,  the  controller  used  the 
wrong  call  sign  in  sequencing  the  aircraft,  resulting  in  the  C-130  continuing  on  its  approach 
in  front  of  the  F-16.  With  the  F-16  converging  very  steep  and  fast  behind  the  C-130,  the 
controller  called  “Cargo  traffic  short  final  on  the  go,”  to  alert  the  F-16  of  the  C-130’s  close 
proximity.  The  controller  assumed  the  F-16  could  clearly  see  the  traffic,  but  the  steep 
approach  angle  placed  the  large  cargo  aircraft  below  the  fighter’s  field  of  view.  Soon  after 
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the  controller’s  radio  call,  the  F- 16  was  cleared  to  land,  and  the  two  aircraft  collided.  The 
two  F-16  pilots  ejected  safely,  but  the  aircraft  impacted  a  Pope  AFB  staging  area  killing 
twenty-four  Army  personnel  and  injuring  more  than  100  soldiers  and  civilians  on  the  ground. 
The  C-130  landed  safely  vdth  significant  damage  (Alkov,  1997). 

This  accident  provides  an  example  of  insufficient  air  traffic  controller  S  A  resulting 
from  increased  workload  in  a  busy  and  complex  airport  traffic  pattern.  (This  will  be 
expanded  on  below  in  defining  the  concept  of  S  A.)  At  the  time  of  this  accident.  Pope  AFB 
was  the  home  to  a  composite  air  wing;  that  is,  the  base  operated  many  different  types  of 
aircraft,  all  with  different  missions,  from  the  same  airfield.  The  disparity  in  aircraft 
performance  parameters  increased  the  cognitive  load  placed  on  controllers  because  of  the 
requirement  to  recall  different  aircraft  capabilities  and  to  coordinate  them  on  this  basis.  The 
NRC  (1998)  has  found  that  decrements  in  SA  may  occur  due  to  factors,  including  high  stress 
and  workload,  and  this  may  increase  the  likelihood  of  ATC  errors.  The  Pope  AFB  accident 
was  found  to  be  the  result  of  inadequate  controller  guidance,  training,  and  experience 
resulting  in  insufficient  controller  SA  (Alkov,  1997). 

Unfortunately,  with  the  volume  and  diversity  of  traffic  at  many  of  the  US  airport’s 
surpassing  capacity  within  the  last  five  years  (Donohue,  2001),  incidents  like  this  are 
becoming  more  likely  in  both  the  military  and  civilian  sectors.  The  workload  and  complexity 
of  ATC  may  ultimately  result  in  decreased  levels  of  air  trafiBc  controller  SA,  which  the  NRC 
(1998,  p.  25)  has  found  to  be  an  underlying  factor  in  complex  system  operator  performance 
and  errors.  Other  research  has  further  described  SA  and  how  it  is  critical  to  ATC.  This  work 
is  reviewed  here. 
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Defining  Situation  Awareness 

Endsley  (1995a)  defines  SA  as  the  “perception  of  the  elements  in  the  environment 
within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the  projection 
of  their  status  in  the  near  future”  (Endsley,  1995a,  p.  36).  In  die  context  of  ATC,  SA  has  also 
been  described  in  many  instances  as  a  picture  of  the  environment  that  a  controller  must 
maintain  internally  (Niessen  et  al.,  1999).  The  three  elements  of  Endsley’s  (1995a)  definition 
delineate  three  different  levels  of  S A: 

•  Level  1 :  Perception  of  elements  in  the  environment, 

•  Level  2:  Comprehension  of  the  current  situation,  and 

•  Level  3:  Projection  of  future  status  (Endsley,  1995a). 

In  regard  to  ATC,  Level  1  SA  is  considered  to  be  perception  of  critical  information 
relating  to  each  aircraft  and  the  environment  within  a  controller’s  airspace.  For  example, 
heading,  altitude,  and  airspeed  of  controlled  aircraft;  the  status  of  each  aircraft,  including  fuel 
on-board,  emergencies,  etc.;  and  airport  and  airspace  information  including  active  runway(s) 
and  airspace  restrictions. 

Level  2  SA  is  considered  to  be  information  consciously  abstracted  from  the 
environment,  based  on  operator  knowledge  of  a  system  (Level  1  SA)  and  the  system  goals 
(e.g.,  in  ATC,  assuring  flight  safety;  Endsley  &  Rodgers,  1994).  For  example,  what  are  the 
aircraft’s  flight  parameters  based  on  its  type  (i.e.,  speed,  range,  weight,  etc.);  what  is  the 
aircraft’s  proximity  to  destinations,  other  aircraft,  and  ground  terrain  based  on  its  current 
coordinates;  and  what  is  the  runway  availability  considering  airborne  and  groimd-based 
activity. 
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Level  3  S  A  is  the  prediction  of  future  aircraft  status  based  on  current  Level  1  and 
Level  2  SA.  Specific  air  traffic  controller  predictions  might  include  future  aircraft  heading, 
altitude,  and  airspeed;  future  aircraft  conflicts;  and  future  proximity  with  other  obstacles.  As 
can  be  discerned  from  this  description  of  SA,  the  levels  of  SA  are  hierarchical  in  that  they 
build  on  each  other.  If  Level  1  SA  is  not  achieved,  it  is  unlikely  that  either  Level  2  or  Level 
3  S A  will  be  achieved.  It  is  important  to  note  that  SA  is  often  referred  to  as  a  snapshot  of 
operator  cognition  in  time,  meaning  an  operator’s  situation  model  is  relevant  for  a  particular 
place  and  time  (Endsley,  1995a). 

In  the  Pope  AFB  accident  described  above,  the  controller  committed  a  Level  1  error 
by  perceiving  the  wrong  information  on  the  C-130,  and  calling  the  aircraft  the  wrong  call 
sign.  Insufficient  training  and  experience  also  contributed  to  the  commission  of  a  Level  2  SA 
error  by  preventing  the  controller  from  accurately  determining  the  F-16’s  speed  and  decent 
rate  based  on  the  knowledge  that  the  F-16  was  performing  an  SI-SFO.  Finally,  assuming  the 
flight  parameters  for  both  aircraft  were  known,  the  controller  failed  to  accurately  predict  their 
future  locations,  a  Level  3  S A  error,  which  prevented  the  controller  from  predicting  the 
intersecting  courses.  Here  it  can  also  be  observed  that  because  of  the  hierarchical  properties 
of  the  levels  of  S  A,  once  a  lapse  in  Level  1  and  Level  2  S  A  occurred,  a  Level  3  error  became 
likely. 

Another  concept  presented  by  Endsley  (1993)  that  is  relevant  to  this  research  is  the 
notion  that  SA  and  workload  may  vary  independently.  Endsley  (1993)  demonstrated  that, 
depending  on  the  task  and  operator  goals,  SA  may  decrease  as  workload  increases  because  of 
the  demand  on  attentional  resources  and  task  saturation,  provoking  incomplete  S  A.  She  also 
observed  that  SA  may  decrease  as  a  result  of  low  workload  and  complacency.  Likewise,  SA 
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may  improve  when  optimal  workload  levels  are  prescribed  for  a  task,  which  provokes 
optimal  arousal  and  performance  (Wickens,  1992).  Relating  to  ATC,  if  a  controller’s 
workload  increases  as  a  result  of  controlling  multiple  aircraft  of  different  types,  requiring 
numerous  communications  and  actions,  then  SA  can  drastically  decrease  as  in  the  case  at 
Pope  AFB.  However,  a  decreased  workload  resulting  from  minimal  traffic  may  also  lead  to 
controller  boredom,  complacency  and  degraded  S  A. 

With  respect  to  the  relation  of  SA  and  performance,  it  is  likely  that  SA  has  a  direct 
impact  on  ATC  functioning  (Endsley  &  Kiris,  1995).  However,  as  Endsley  (1995a)  pointed- 
out,  SA  does  not  guarantee  good  performance,  but  it  is  “a  factor  that  will  increase  the 
probability  of  good  performance”  (Endsley,  1995a,  p.  40). 

Wickens  (2002)  also  breaks  SA  down  into  three  components  and  identifies  the 
aspects  and  features  of  aviation  systems  that  are  most  relevant  to  each  component.  Wickens’ 
(2002)  components  include:  spatial  awareness,  system  awareness,  and  task  awareness. 
Three-dimensional  spatial  awareness  in  an  ATC  context  is  the  awareness  of  all  aircraft 
locations  and  orientations  within  the  controlled  area.  Each  of  these  parameters  are  dynamic 
and  have  an  effect  on  the  others.  For  example,  if  an  aircraft’s  orientation  is  changed  (e.g.,  it 
pitches  nose-down)  then  the  future  location  of  that  aircraft  will  change  as  well  (i.e.,  it  will 
likely  be  lower)  and  it  may  have  new  flight  parameters  (e.g.,  a  possible  increase  in  airspeed). 
There  may  also  be  unpredictable  time  lags  between  these  variables,  increasing  then- 
complexity. 

The  second  component  of  SA,  system  awareness  (or  mode  awareness),  is  knowing  the 
status  of  the  stimuli  in  the  environment,  or  being  aware  of  the  flight  status  of  each  aircraft. 
Wickens  (2002)  clarifies  this  by  saying,  “people  remember  actions  that  they  themselves  have 
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initiated  better  than  those  initiated  by  another  agent”  (p.  131).  In  the  Pope  AFB  example,  the 
C-130  continued  to  final  approach  without  direction  firom  the  controller,  because  the 
controller  used  the  wrong  call  sign.  As  a  result  of  the  controller  not  initiating  correct  actions, 
the  controller  lost  awareness  about  the  C-130’s  location  and  intentions.  Automation  can  have 
a  significant  impact  on  system  awareness,  because  the  more  a  controller  is  removed  fi'om  the 
control  loop  the  less  system  awareness  is  retained  (Wickens,  2002). 

Finally,  Wickens  (2002)  defines  task  awareness  as  knowing  what  tasks  are  required  to 
achieve  the  overarching  goal  and  the  order  in  which  to  address  the  tasks.  Knowledge  of  a 
system’s  goals  allows  an  operator  to  respond  to  a  situation  quickly  and  efficiently  even  if  an 
operator  has  not  obtained  sufficient  SA.  For  example,  an  air  traffic  controller  may  not  be 
consciously  thinking  about  the  fact  that  there  is  a  general  aviation  aircraft  traversing  a  tower 
controlled  airspace  without  the  intent  to  land,  but  as  another  aircraft  approaches  the  airport  to 
land,  the  controller’s  goal  of  aircraft  separation  causes  the  controller  to  quickly  and 
accurately  separate  the  two  vehicles. 

Like  Wickens  (2002),  Niessen  et  al.  (1999)  state  that  there  are  three  requirements  for 
the  ATC  picture,  or  controller  SA,  including; 

(1)  a  cyclical  update  of  varying  relations  between  aircraft  and  the  environment  as  a 
basis  for  SA; 

(2)  the  prediction  of  the  future  states  in  order  to  detect  risky  relationships  between 
aircraft;  and 

(3)  the  determination  of  the  temporal  sequence  of  cognitive  activities  in  meeting 
simultaneous  task  requirements  (Niessen  et  al.,  1999). 
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Niessen  et  al.  (1999)  also  presented  a  model  that  depicted  the  different  phases  of 
controller  information  processing  required  for  S  A,  including  data  selection,  anticipation, 
conflict  resolution,  update  and  control.  These  five  steps  contribute  to  the  mental  picture 
developed  by  the  controller.  In  addition,  according  to  Niessen  et  al.  (1999)  these  steps  do  not 
occur  in  parallel,  but,  rather,  must  be  performed  separately.  Experience  also  plays  a  large 
roll  in  the  use  of  this  model,  as  experienced  operators  “use  less  information,  but  that  this 
information  is  diagnostically  more  relevant”  (Niessen  et  al.,  1999,  p.  1513). 

Situation  Awareness  and  Other  Cognitive  Constructs 

There  are  many  cognitive  constructs  that  have  been  hypothesized  to  have 
relationships  with  SA  and  that  may  be  essential  to  complex  system  operator  achievement  of 
S A,  including  attention,  perception,  and  memory.  Attention,  as  discussed  by  Endsley 
(1995a),  is  a  critical  factor  in  developing  SA  and  is  needed  for  perception,  information 
storage,  decision  making,  and  response  execution  functions  as  part  of  human  information 
processing  (HIP).  Attention  is  required  to  filter  information  in  an  environment  that  is 
essential  for  the  development  of  SA  fiom  other  distracter  information.  Attentional 
mechanisms  are  used  in  identifying  elements  in  the  environment  that  are  familiar,  or 
represent  habituated  stimuli  (Cowan,  1988),  compared  to  novel  stimuli  that  may  require 
further  processing.  This  filtering  of  information  allows  complex  system  operators  to  manage 
large  amounts  of  data  in  the  environment.  For  example,  it  may  be  critical  to  air  traffic 
controllers  in  achieving  S A  even  when  many  different  aircraft  are  operating  within  their 
airspace. 
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Perception  is  another  cognitive  construct  that  is  considered  essential  for  achieving  SA 
(within  ATC).  Endsley  (1995a)  states  that  working  memory  is  the  primary  aspect  of 
cognition  used  in  recognition/classification  of  stimuli  in  an  environment.  Working  memory 
also  serves  as  a  dynamic  storage  for  concurrent  operator  task  goals.  The  air  traffic  controller 
compares  his  or  her  current  goals  with  perceived  information  in  order  to  comprehend  current 
traffic  situations  (Level  2  SA).  In  addition,  perception  of  stimuli  within  the  environment 
(facilitated  by  recognition  and  classification  through  working  memory)  leads  to  the 
development  of  an  internal  situation  model  that  is  matched  to  mental  models  in  long-term 
memory.  This  process  forms  the  basis  for  decision  making  and  projection  of  future  states  of 
a  system  (Level  3  SA). 

Long-term  memory  is  related  to  SA  in  that  situation  models  may  be  used  to  enhance 
and  expand  existing  mental  models.  This  may  allow,  for  example,  expert  controllers  to 
effectively  sort  information  and  determine  which  stimuli  are  most  important  at  a  particular 
moment.  Therefore,  the  controller  can  achieve  adequate  SA  in  a  stimulus-saturated 
environment.  This  suggests  a  circular  relationship  of  situation  models  and  mental  models. 
Long-term  memory  can  also  effect  the  efficiency  (i.e.,  speed  and  accuracy)  of  perceiving 
information  in  the  environment  (Level  1  SA).  Expert  controller’s  expectations  and 
preconceptions  about  task  information  based  on  long-term  memories  (mental  models)  allow 
accurate  Level  1  S A  to  be  achieved  through  timely  aircraft  and  environmental  perception. 
(Endsley,  1995a) 
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Identifying  Goals  and  Tasks  for  Air  Traffic  Control 

Endsley  and  Rodgers  (1994)  and  Endsley  and  Jones  (1995)  conducted  research  to 
determine  the  goals  and  tasks  for  ATC,  as  well  as  controller  SA  requirements,  and  the 
information  required  to  accomplish  ATC  goals  by  using  Goal  Directed  Task  Analysis 
(GDTA).  Endsley  and  Rodgers  (1994)  interviewed  retired  air  traffic  controllers  and  analyzed 
simulated  ATC  scenarios  to  develop  a  very  extensive  list  of  SA  requirements  for  en  route 
controllers.  En  route  ATC  refers  to  the  control  centers  across  the  country  responsible  for 
separating,  navigating,  and  handing-off  aircraft  between  airport,  or  terminal,  air  space 
(Nolan,  1999).  These  en  route  requirements  consist  of  overarching  ATC  goals.  Secondly, 
subgoals  are  defined  which  are  required  for  accomplishing  the  goals.  Subsequently, 
questions  that  must  be  answered  by  the  controller  to  accomplish  subgoals  are  developed. 
These  questions  can  later  be  used  to  assess  SA  in  the  task,  and  will  be  discussed  in  a  later 
section  of  this  document.  Finally,  the  high-level  and  low-level  SA  requirements  for 
answering  questions  are  outlined. 

Endsley  and  Jones  (1995)  performed  a  similar  analysis  to  evaluate  the  SA 
requirements  for  Terminal  Radar  Approach  Control  (TRACON)  centers.  A  TRACON  area 
refers  to  the  controlled  airspace  within  the  immediate  vicinity  of  a  congested  airport  (Nolan, 
1999).  A  portion  of  one  GDTA  conducted  by  Endsley  and  Jones  (1995)  for  avoiding  aircraft 
conflicts  during  TRACON  operations  (i.e.,  separate  aircraft)  is  presented  here  with  labels 
differentiating  the  various  elements  of  the  GDTA: 
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[goal]  1.1  Separate  aircraft 
[subgoal]  1.1,1  Assess  aircraft  separation 

[question]  -  vertical  separation  meets  or  exceeds  limits? 

—  vertical  distance  between  aircraft  along  route  (projected) 

—  vertical  distance  between  aircraft  (current) 

—  aircraft  altitude  (current) 

“  altitude  accuracy 

—  altitude  (assignment) 

—  altitude  rate  of  change  (climbing/descending) 

(Endsley  &  Jones,  1995,  p.  19) 

This  type  of  analysis  is  an  important  tool  for  understanding  ATC  SA  requirements,  and  is 

valuable  for  designing  and  developing  future  ATC  systems  by  considering  controller 
information  needs. 


[high-level  SA  information] 
[high-level  SA  information] 
[low-level  SA  information] 
[low-level  SA  information] 
[low-level  SA  information] 
[low-level  SA  information] 


Situation  Awareness  and  Air  Traffic  Control  Performance  and  Automation 

All  definitions  of  S  A  agree  that  the  mental  construct  has  a  critical  impact  on  human 
performance,  particularly  in  a  complex  system  like  ATC.  Air  traffic  control  is  a  very  unique 
and  demanding  occupation  in  terms  of  the  magnitude  of  information  presented  in  multiple 
mediums,  the  requirements  to  make  timely  decisions,  and  the  importance  of  correct 
decisions.  It  is  imperative  in  ensuring  flight  safety  that  air  traffic  controllers  are  assigned 
appropriate  levels  of  workload  and  achieve  adequate  levels  of  S  A.  In  the  Pope  AFB  incident, 
the  controller  had  relatively  high  levels  of  workload  and  insufficient  SA  due  to  inadequate 
experience  and  training,  resulting  in  performance  decrements  (e.g.,  forgetting  call  signs  and 
incorrect  clearances). 
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One  method  explored  to  alleviate  stress  and  workload  on  controllers  is  the  use  of 
automation  in  ATC.  In  the  following  sections,  definitions  and  forms  of  automation  are 
provided  in  addition  to  potential  advantages  and  disadvantages  of  using  automation  within 
ATC. 

1.2  Automation  in  Air  Traffic  Control 

With  the  increase  in  aircraft  volume  and  the  types  of  aircraft  being  flown  in  airspaces 
around  the  world,  actions  must  be  taken  in  order  to  help  controllers  to  manage  the  inevitable 
airspace  congestions  while  maintaining  safety,  efficiency,  and  reducing  costs  (NRC,  1998). 
One  common  strategy  for  relieving  controller  stress  is  partitioning,  or  reducing  the  airspace 
an  individual  controller  is  responsible  for.  However,  according  to  Hopkin  (1999)  partitioning 
has  reached  its  maximum  obtainable  benefits  as  the  airspace  becomes  so  small  that  a 
controller  is  responsible  for  more  frequent  hand-offs  of  aircraft  to  other  controllers,  more 
communications,  and  increased  coordination;  thus,  further  increasing  woridoad  and  possibly 
decreasing  SA.  As  technology  provides  more  capabilities  to  ATC,  a  more  feasible  solution 
to  remedy  increased  airspace  congestion  is  to  allocate  some  of  the  controller’s 
responsibilities  to  computers,  or  automation. 

Defining  Automation 

Automation  has  been  defined  as,  “the  execution  by  a  machine  agent  of  a  function 
previously  carried  out  by  a  human”  (Parasuraman  &  Riley,  1997,  p.  231).  Furthermore,  the 
NRC  (1997)  expands  this  definition  by  including  either  partial  or  full  automation  of  a 
function  that  was  previously  (partially  or  fully)  accomplished  by  a  human.  Hopkin  (1999) 
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defines  system  fimctions  that  are  fiilly  automated  as,  “fimctions  that  do  not  require,  and  often 
do  not  permit,  any  direct  human  participation  or  intervention  in  them”  (p.  498).  Billings 
(1997)  further  describes  automation  as  a  tool  that  allows  a  human  operator  to  accomplish  a 
task  with  less  effort,  or  to  complete  a  task  that  would  have  been  impossible  otherwise. 

Air  traffic  control  automation  first  appeared  in  the  1960’s  in  the  form  of  the 
Automated  Radar  Terminal  System  (ARTS),  which  incorporated  all  aircraft  information 
within  one  controlled  space  into  one  radar  display.  Modem  versions  of  the  ARTS  are  still  in 
place  today.  Since  the  1960’s,  many  new  applications  have  been  developed  for  automation 
integration.  These  will  be  discussed  in  the  following  sections. 

Identifying  Forms  of  Automation 

Wickens  (1992,  p.  531-532),  in  attempting  to  define  different  types  of  automation, 
identified  three  general  circumstances  in  which  automation  can  be  employed.  First, 
automation  can  be  used  to  perform  functions  that  the  human  operator  cannot  perform  because 
of  inherent  limitations.  For  example,  humans  are  generally  limited  in  their  ability  to 
“monitor  for  prespecified  events,  store  and  retrieve  coded  information,  and  perform  repetitive 
activities  reliably”  (Sanders  &  McCormick,  1993,  p.  730-731).  These  are  activities  which 
computers  are  particularly  apt  at  performing  (Wickens  &  Hollands,  2000).  Examples  of  this 
form  of  automation  in  ATC  include  flight  data  processing  (FDP)  for  en  route  centers  where 
information  about  flights  and  controller  interactions  is  stored  in  a  database  for  later  data 
synthesis,  data  presentation,  and  computations  to  be  used  for  performance  evaluation  and 
training.  These  are  all  processes  that  humans  perform  poorly  compared  to  computers 
(Wickens  &  Hollands,  2000).  Another  example  of  this  form  of  automation  is  smoothing  of 
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aircraft  flight  information  on  ATC  displays.  Flight  data  regarding  specific  aircraft  is 
gathered  incrementally  and  presented  to  controllers  on  displays,  but  this  unprocessed 
information  presents  jagged,  irregular  display  movements,  which  can  be  distracting  to 
controllers  without  the  aid  of  an  automated  process  called  smoothing  that  extrapolates  the 
flight  data  into  smooth,  fluid  movements  of  the  aircraft  displays  (NRC,  1997). 

Wickens  (1992)  also  describes  automation  used  to  perform  functions  that  humans 
perform  poorly  or  at  the  cost  of  high  workload.  As  previously  mentioned,  as  airspace 
becomes  more  congested,  and  partitioning  decreases  the  sizes  of  airspace,  controllers  are 
required  to  perform  the  taxing  process  of  handing-off  aircraft  to  other  controllers  more 
firequently.  Automated  systems  have  been  implemented  to  handle  hand-offs,  thereby 
eliminating  the  need  for  verbal  communication  between  controllers  as  aircraft  pass  through 
airspace,  and  decreasing  both  controller  and  pilot  workload.  In  addition,  controllers  were 
previously  required  to  obtain  aircraft  altitude,  speed,  and  other  information  verbally  fi-om 
pilots  for  hand-offs.  They  were  also  expected  to  perform  the  complex  process  of  determining 
specific  aircraft  headings  based  on  limited  and  inaccurate  aircraft  information  presented  on 
controller  displays.  This  information  including  aircraft  call  sign,  type  of  aircraft,  destination 
airport,  navigational  fixes,  altitude,  and  ground  speed  are  all  now  automatically  presented  to 
the  controller  in  an  accurate  data  tag  embedded  in  the  radar  display  (NRC,  1997). 

Automation  is  also  used  to  augment  or  assist  humans  in  areas  they  show  performance 
limitations  (Wickens,  1992).  In  an  effort  to  aid  controllers  with  automation,  the  Federal 
Aviation  Administration  (FAA)  introduced  the  center-TRACON  automation  system  (CTAS). 
This  system  assists  controllers  in  successfully  clearing  aircraft,  rather  than  implementing 
fully  automated  systems  (Dillingham,  1998).  Other  ATC  automation  systems  utilized  to 
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assist  controllers  are  the  Minimum  Safe  Altitude  Warning  (MSAW)  system,  the  Conflict 
Alert  (CA)  system,  and  the  automated  (aircraft)  track  deviation  system  (NRC,  1997).  In 
addition,  controller  requirements  to  maintain  aircraft  separation  from  other  aircraft  and  the 
ground  has  been  augmented  by  various  forms  of  cockpit  automation  systems  including  the 
Traffic  Alert  and  Collision  Avoidance  System  (TCAS)  and  the  Groimd  Proximity  Warning 
System  (GPWS).  These  systems  alleviate  controller  workload  by  allocating  some  separation 
responsibilities  to  pilots  (NRC,  1997;  Nolan,  1999). 

Advantages  and  Disadvantages  of  Automation 

Automation  can  have  a  profound  effect  on  human  performance.  Automation  has 
many  advantages  and,  with  the  availability  of  modem  technology,  has  become  a  viable 
option  for  reducing  the  load  on  air  traffic  controllers,  increasing  ATC  efficiency  and  safety, 
and  decreasing  cost.  However,  when  improperly  implemented,  there  are  also  many  potential 
disadvantages  of  using  automation. 

Workload 

One  of  the  primary  goals  of  incorporating  automation  into  a  system  is  often  to 
decrease  workload.  Laois  and  Giannacourou  (1995)  conducted  research  to  determine  the 
impact  of  five  currently  automated  ATC  systems,  including:  Electronic  Data  Display  (EDD), 
Trajectory  Prediction  Aids  (TP A),  Conflict  Detection  Aids  (CD A),  Data  Links  Applications 
(DL),  and  Clearance  Advisory  Aids  (CAA).  Their  study  consisted  of  observations, 
interviews,  and  presenting  expert  controllers  with  questionnaires  pertaining  to  the  automated 
systems.  They  determined  that  all  five  forms  of  automation  increased  formalization  (of 
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communication),  performance,  and  flexibility  while  decreasing  workload  by  assuming 
responsibility  for  ATC  functions.  They  also  determined  that  “significant  workload 
reductions  will  be  effected  by  aiding  decision  making  and  predictive  activities  and  not  by 
automation  of  routine  data  acquisition  and  communication  activities”  (Laois  & 

Giannacourou,  1995,  p.  395). 

However,  introducing  automation  does  not  always  decrease  workload;  in  some  cases, 
automation  may  actually  increase  workload.  Parasuraman  and  Byrne  (2002)  suggest  three 
situations  where  improper  automation  may  increase  operator  workload.  First,  if  automation 
requires  extensive  reprogramming  or  data  entry  during  critical  events,  then  the  automation 
may  actually  increase  workload  (e.g.,  if  a  controller  is  required  to  input  data  into  a  system 
during  a  close  proximity  warning).  Second,  if  the  nature  of  the  automated  system  requires 
the  user  to  continuously  evaluate  the  benefit  of  automation  compared  to  manual  control  (as  in 
adaptively  automated  systems),  the  system  may  induce  higher  workload.  This  is  important  in 
the  implementation  of  adaptively  automated  systems  (Clamann  &  Kaber,  in  press).  Finally, 
if  the  results  of  automation  are  so  critical  or  unreliable  that  the  automation  performance  must 
be  continually  evaluated  for  errors,  user  workload  may  increase.  This  is  a  critical  issue  for 
ATC  automation,  because  ATC  errors  can  be  catastrophic.  Riley  (1996)  foxmd  that  in 
situations  where  automation  significantly  increased  workload,  given  the  choice,  operators 
often  chose  to  operate  manually. 

System  Trust 

Operator  trust  in  an  automated  system  is  another  important  automation  factor  in  ATC 
that  is  affected  by  the  design  and  performance  of  automation  (Wickens,  2000).  The  NRC 
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(1998)  presented  seven  characteristics  that  determine  trust  in  an  automated  system: 
reliability,  robustness,  familiarity,  imderstandability,  explication  of  intentions,  usefulness, 
and  dependence.  Any  or  all  of  these  factors  can  contribute  to  one  of  many  resulting 
scenarios:  overtrust,  mistrust,  and  any  level  of  trust  between  the  two  (Parasuraman  &  Riley, 
1997).  The  primary  factor  influencing  overtrust  is  automation  reliability  (Wickens,  2000; 
Parasuraman  &  Byrne,  2002).  If  a  system  appears  to  be  extremely  reliable,  and  errors  are 
very  rare,  then  the  operator  will  place  a  great  degree  of  trust  in  the  system  (Parasuraman  & 
Riley,  1997).  However,  perfect  automation  performance  is  rare,  and  too  much  trust  also 
fosters  complacency,  which,  as  Endsley  (1993)  noted,  can  decrease  SA  and  make  system 
recovery  from  errors  much  more  difficult. 

Lower  or  variable  levels  of  system  reliability  can  be  used  to  overcome  overtrust 
problems  (Parasuraman  et  al.,  2000),  but  a  very  low  level  of  reliability  can  lead  to  mistrust. 
Mistrust  often  leads  to  operators  refraining  from  use  of  the  automation  or  ignoring  it. 
Mistrust  has  also  been  attributed  to  an  unreliable  system  with  a  high  level  of  false  alarms 
(Parasuraman  &  Riley,  1997;  Wickens,  2000).  In  a  system  like  ATC,  errors  have  great 
consequences  (e.g.,  aircraft  accidents  may  result  in  fatalities,  aircraft  damage  is  very 
expensive,  and  delays  are  very  costly).  Therefore,  system  designs  often  aim  to  decrease  the 
likelihood  of  a  missed  error,  at  the  cost  of  increasing  the  probability  of  a  false  alarm 
(Wickens,  1992).  In  addition  to  false  alarms,  a  lack  of  understandability  of  automation  may 
also  lead  to  mistrust  (Endsley,  1996).  The  aircraft  Flight  Management  System  (FMS),  a 
cockpit  computer  which  aids  pilots  in  aircraft  navigation  and  communication,  and  the 
Standard  Terminal  Automation  Replacement  System  (STARS),  a  terminal  system  that  is 
currently  being  employed  by  ATC  to  integrate  aircraft  hand-off,  communication,  radar, 
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weather,  and  air  traffic  data,  are  both  very  complex  systems  (Nolan,  1999).  With  an 
incomplete  understanding  of  all  of  their  various  modes  and  behaviors,  pilots  and  controllers 
may  not  trust  their  capabilities.  Automation  design  and  training  is  responsible  for  making 
complex  automation  systems  transparent  and  understandable  in  order  to  decrease  mistrust. 

Out-of-the-Loop  Unfamiliarity 

Another  possible  drawback  of  implementing  automation  is  reducing  the  level  of 
operator  interaction  with  the  system  and  reducing  familiarity,  which  Wickens  (1992)  defines 
as  “out-of-the-loop  unfamiliarity”  (OOTLUF).  As  more  ATC  fimctions  and  operations  are 
performed  by  machines,  controllers  have  less  interaction  with  the  system  and  may  become 
less  aware  of  system  operations.  Parasuraman,  Sheridan,  and  Wickens  (2000)  point  out  that 
automation  may  change  the  role  of  humans  fi-om  system  controllers  to  supervisors 
monitoring  system  status,  a  fimction  to  which  humans  are  not  well  suited  (Endsley,  1995b). 
Out-of-the-loop  (OOTL)  performance  may  reduce  a  controller’s  ability  to  detect  that  a 
problem  has  occurred,  determine  the  current  state  of  the  system,  understand  what  has 
happened  and  what  courses  of  actions  are  needed  (develop  SA),  and  react  to  the  situation 
(Endsley,  1996). 

Endsley  and  Kaber  (1999)  identified  five  consequences  when  implementing 
automation  in  a  manner  that  places  the  controller  OOTL,  including:  difficulty  in  assuming 
control  when  needed,  vigilance  decrements,  complacency  (discussed  previously),  skill  decay, 
and  loss  of  SA.  With  respect  to  the  first  consequence  associated  with  implementing 
automation,  an  operator  may  be  removed  from  a  control  loop  to  the  extent  that  in  the  event  of 
an  emergency  or  automation  feilure,  the  controller  may  not  be  able  to  regain  manual  control. 
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Laois  and  Giannacourou  (1995)  observed  this  in  their  study  of  expert  controllers  when  they 
found  that  automation  which  places  the  controller  OOTL  can  result  in  the  inability  to  react  to 
an  emergency  situation. 

Vigilance  decrements  are  the  result  of  extended  periods  of  monitoring,  which 
decrease  operator  sensitivity  to  task  stimuli,  so  that  the  ability  of  the  user  to  discriminate, 
recognize,  and  diagnose  critical  events  is  compromised  (NRC,  1997),  and  the  accuracy  and 
speed  of  these  responses  can  be  severely  affected.  An  ATC  example  of  potential  vigilance 
decrements  can  by  foimd  during  the  infrequent,  yet  critical,  instance  that  a  data  tag  on  a  radar 
display  is  inaccurate.  A  controller  is  always  monitoring  data  for  accuracy,  but  when  the 
controller  is  removed  from  the  active  process  of  gathering  aircraft  data,  and  this  information 
is  presented  automatically,  a  vigilance  decrement  can  occur  resulting  in  issuance  of  a  wrong 
clearance  or  a  clearance  to  the  wrong  aircraft  because  of  inaccurate  data.  Sufficient 
controller  breaks  and  active  involvement  in  system  control  during  low  workload  periods  may 
decrease  the  likelihood  of  vigilance  decrements  (NRC,  1997). 

Complacency,  as  discussed  earlier,  is  a  function  of  trust  within  a  system,  reliance  on 
the  automation,  and  confidence  in  the  automation.  Trust  in  automation  is  critical  for 
effective  system  monitoring,  but  too  much  trust  can  result  in  complacency,  leading  to  errors 
and  missed  critical  events  (Parasuraman  &  Riley,  1997).  Endsley  (1993)  suggested  that  low 
levels  of  workload  can  also  induce  complacency,  and  that  complacency  can  negatively  affect 
S  A,  making  ftie  operator  less  aware  of  system  status. 

Complacency  is  a  difficult  issue  to  remedy  when  operators  are  removed  from  a 
system  control  loop  (Laois  &  Giannacourou,  1995).  Parasuraman  et  al.  (2000)  found  that  if 
automated  system  reliability  was  variable,  alternating  between  high  and  low  levels,  then 
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operator  trust  in  the  system  also  became  variable,  decreasing  complacency.  Salient  warnings 
and  alarms,  and  providing  information  regarding  system  status  may  also  reduce  complacency 
and  stimulate  operator  attention  by  alerting  them  to  system  failures  (Parasuraman  &  Riley, 
1997;  NRC,  1998).  Another  method  for  overcoming  complacency  issues  is  the  use  of  more 
advanced  forms  of  automation,  discussed  in  Section  1.3. 

If  controllers  work  OOTL  for  extended  periods  of  time,  or  automated  systems  are 
highly  reliable  and  evoke  high  levels  of  operator  trust,  controller  skills  may  degrade  (NRC, 
1999).  This  drawback  of  automation  is  known  as  skill  decay  (Endsley  &  Kaber,1999).  In  the 
automation  of  most  systems,  skill  decay  is  inevitable  as  an  operator  becomes  less  actively 
involved  in  control.  For  example,  in  a  highly  automated  system  like  CTAS  or  STARS, 
controllers  are  likely  to  lose  the  skills  required  to  perform  basic  ATC  functions  such  as  flight 
tracking  or  groimd  and  aircraft  close  proximity  detection  (NRC,  1998).  This  becomes 
problematic  when  a  situation  requires  a  controller  to  revert  to  manual  control,  as  discussed 
earlier.  Recurring  training,  periods  of  manual  control,  and  thoughtful  automated  system 
design  are  important  for  maintaining  controller  skills  (NRC,  1998). 

Situation  Awareness 

Automation  can  have  a  profound  effect  on  SA.  As  discussed  earlier,  automation  may 
remove  the  operator  from  the  role  of  active  system  controller  to  the  role  of  monitoring,  or 
supervisory  controller.  Although  this  may  decrease  workload  (or  increase  it  in  some 
situations),  the  operator  may  lose  the  ability  to  perceive,  comprehend,  or  project  the  future 
status  of  the  system  within  the  operating  environment,  or  a  loss  of  S  A  (Wickens,  2002). 
Endsley  (1996)  identified  SA  as  the  most  challenging  aspect  of  monitoring  within  an 
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automated  system,  and  she  said  that  maintaining  SA  is  vital  for  proper  decision  making  and 
performance.  She  also  states  that  partly  or  fully  automated  tasks  can  decrease  operator  S A 
compared  to  fully  manual  control,  because  having  a  passive  role  in  the  system’s  performance 
prevents  the  user  from  understanding  its  current  and  future  status.  Fmthermore,  “turning  a 
human  operator  from  a  performer  into  an  observer  can,  in  and  of  itself,  negatively  affect  SA” 
(Endsley,  1996,  p.  3).  These  SA  deficiencies  become  apparent  when  a  supervisory  controller 
is  required  to  initiate  and  maintain  manual  control  of  a  system  in  response  to  an  automation 
failure.  Automation  of  different  levels  of  HIP  can  also  affect  SA  in  different  ways.  Endsley 
and  Kiris  (1995)  and  Kaber,  Onal,  and  Endsley  (1999)  determined  that  low-level  automation 
can  improve  SA,  while  high-level  automation  like  decision-making  aids  can  decrease  SA. 

Examples  of  ATC  automation  leading  to  decrements  in  controller  SA  can  be  found  in 
the  many  different  types  of  mode  errors  that  occur  with  automated  ATC  systems.  A  mode 
refers  to  “the  setting  of  a  system  in  which  inputs  to  the  system  result  in  outputs  specific  to 
that  mode  but  not  to  other  modes”  (Parasuraman  &  Byrne,  2002,  p.  328).  Mode  errors  result 
when  an  operator,  because  of  high  levels  of  automation  and  multiple  system  modes,  is 
unaware  of  the  mode  in  which  a  system  is  currently  operating;  thus,  the  controller  inputs 
data,  or  extrapolates  data  from  a  system,  assuming  the  wrong  system  mode  is  active.  High 
level  automation  can  decrease  SA  and  the  operator  becomes  imaware  of  mode  changes  and 
activities  prior  to  interacting  with  the  system.  Many  aircraft  accidents  have  resulted  from 
low  levels  of  SA  and  mode  errors,  particularly  with  the  aircraft  FMS,  because  the  pilot  was 
OOTL  (Parasuraman  &  Byrne,  2002;  NRC,  1997). 

All  of  these  problems  can  be  attributed  to  operators  performing  tasks  while  OOTL. 
However,  there  are  aspects  of  automation  design  that  can  be  manipulated  to  maintain 
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operator  S  A  by  maintaining  in-the-loop  performance.  One  aspect  of  system  automation 
critical  to  SA  is  appropriate  feedback.  Proper  feedback  about  a  system’s  status  can  improve 
controller  SA  (Endsley,  1996),  improve  OOTL  performance,  and  improve  safety  in 
automated  systems  (Parasuraman  &  Byrne,  2002).  In  the  example  of  mode  errors,  proper 
feedback  about  the  current  system  mode  changes  can  improve  operator  SA  and  decrease 
OOTL  errors.  In  addition,  Parasuraman  and  Byrne  (2002)  suggest  that  automated  states  and 
behaviors  be  salient  and  imderstandable  for  better  automated  performance.  This  includes 
warnings  and  signals  about  the  system  status  (e.g.,  feedback)  being  obvious  and  clear  in  their 
meaning. 

Summary 

Air  traffic  control  automated  systems  are  not  perfect.  Although  their  intention  is  to 
alleviate  workload  placed  on  controllers  and  improve  system  performance,  unfortunately  the 
manner  in  which  automated  systems  have  been  designed  and  implemented  has  sometimes  led 
to  the  opposite.  There  are  other  automation  methods  that  focus  on  the  design  and  integration 
of  modem  technologies  with  hiunan  control.  As  research  and  development  make  these  forms 
of  automation  more  accessible  and  exposes  the  advantages  of  using  new  methods,  they  may 
become  actual  approaches  for  overcoming  some  of  the  disadvantages  of  existing  automation. 

1.3  Advanced  Automation  in  Air  Traffic  Control 

The  drawbacks  of  automation,  which  have  been  discussed,  are  primarily  relevant  to 
systems  that  operate  under  either  full  automation  or  forms  of  supervisory  control.  However, 
Hopkin  (1998)  suggests  four  broad  forms  of  automation  that  can  be  used  in  ATC  systems. 
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some  of  which  extend  beyond  the  previously  discussed  forms  of  traditional  automation, 
including; 

•  human  tasks  that  can  be  automated  completely  so  the  human  retains  no  role  in  them; 

•  human  tasks  that  can  be  performed  with  various  forms  of  computer  assistance  so  that 
they  may  be  simplified,  reduced  in  procedure,  reduced  in  detail  or  frequency,  or 
permit  the  controller  to  act  more  quickly; 

•  human  tasks  that  can  be  divided,  leaving  some  parts  for  the  human  to  perform  while 
other  parts  are  performed  automatically;  and 

•  human  roles  that  can  be  changed  from  tactical  interventions  during  flights  of  single 
aircraft  to  strategic  preplanning  of  traffic  flows. 

Full  Automation 

The  first  form  of  automation  represents  traditional  technology-centered  automation 
and  is  typical  of  early  automated  systems.  The  Full  Service  Level  (FSL)  is  a  feature  of 
STARS  that  fully  automates  the  ATC  data  gathering  process.  Radar,  flight  plan,  and  weather 
information  was  once  collected  manually  by  controllers,  one  element  at  a  time,  and  the 
controller  was  required  to  integrate  this  information  for  proper  aircraft  clearance.  The  FSL 
automatically  integrates  aircraft  radar  images,  flight  plans,  and  pertinent  weather  information 
and  presents  it  to  the  controller  in  an  easy  to  discern  display  (NRC,  1998).  However,  this 
type  of  data  gathering  automation  may  remove  the  controller  from  the  control  loop,  and 
result  in  negative  automation  consequences. 
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Computer  Assistance 

The  CTAS  is  a  good  example  of  an  ATC  automated  system  which  provides 
automation  assistance  for  controllers.  The  CTAS  has  four  components,  the  first  being  the 
Traffic  Management  Advisor  (TMA).  This  computer  system  gathers  information  about  all  of 
the  aircraft  inbound  for  a  landing,  and  uses  the  aircraft  parameters  to  develop  a  plan 
including  a  sequence  and  schedule  for  landing  all  of  the  aircraft.  This  plan  may  be  accepted 
or  adjusted  by  the  controller  to  meet  specific  requirements  not  considered  by  the  automated 
system  (e.g.,  emergency  procedures,  special  requests,  etc.;  NRC,  1998).  Secondly,  the  CTAS 
system  incorporates  a  Descent  Advisor  (DA)  which  uses  aircraft  type,  capabilities, 
atmospheric  conditions,  and  a  plan  from  the  TMA  to  advise  controllers  about  decent  rates, 
speeds,  and  durations  for  each  aircraft  (Hilbum,  Joma,  Byrne,  &  Parasuraman,  1997). 

Lastly,  the  Final  Approach  Spacing  Tool  (FAST)  provides  controllers  with  aircraft  sequence, 
runway  assignments,  speed,  and  heading  advisories.  The  final  approach  portion  of  a  flight  is 
the  most  crucial  segment  given  the  close  proximity  of  aircraft  near  an  airport,  limited  time  to 
make  decisions,  and  the  precision  and  safety  requirements  of  decisions  (NRC,  1997).  The 
FAST  has  the  capability  to  quickly  and  precisely  adjust  to  dynamic  situations  during  the  final 
approach  segment.  The  DA  and  FAST  advisories  are  presented  automatically  to  the 
controller  through  the  aircraft  data  tags  on  the  radar  display.  The  Expedite  Departure  Path 
(EDP)  program  is  a  recent  addition  to  the  CTAS  which  incorporates  the  same  capabilities  of 
the  FAST  and  DA,  but  for  directing  aircraft  that  are  departing  a  controlled  area.  In  addition, 
these  capabilities  are  becoming  available  not  only  for  the  aircraft  arriving  and  departing  the 
controlled  airport,  but  for  aircraft  arriving  and  departing  smaller  airports  within  close 
proximity  to  the  controlled  airport  (NRC,  1998;  Nolan,  1999). 
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Function  Allocation 


Hopkin  (1998)  also  suggests  dividing  ATC  tasks  between  human  and  computer 
controllers  as  a  method  for  effective  automation.  Sanders  and  McCormick  (1993)  define  this 
approach  to  automation  as  function  allocation  (FA).  The  primary  dilemma  when  applying 
FA  to  an  automated  system,  is  deciding  which  functions  should  be  automated  and  which 
functions  should  be  performed  manually  by  the  operator.  In  general,  Kaber  and  Endsley 
(1997)  determined  that  “in  dynamic,  multi-task  environments  some  human  manual 
performance  is  useful  to  overall  system  functioning”  (p.  209),  and  that  shared  human  and 
automated  control  of  a  system  is  often  better  than  fully  automated  control.  Studies  have 
investigated  the  allocation  of  automated  assistance  to  the  various  stages  of  human-machine 
system  information  processing  in  dynamic  control  tasks  to  determine  which  forms  of 
automation  are  most  advantageous  for  supporting  human  performance  (Kaber  &  Endsley, 
1997;  Parasuraman  et  al.,  2000;  Clamann,  Wright,  &  Kaber,  2002).  Kaber  and  Endsley 
(1997)  defined  10  different  forms,  or  levels,  of  automation  (LOAs)  ranging  from  manual 
control  to  full  automation  by  developing  FA  schemes  for  humans  and  machines  in  dynamic 
control  tasks.  Their  empirical  research  determined  that  the  LOA  has  a  significant  effect  on 
radar  monitoring  and  target  elimination  task  performance.  Specifically,  they  foimd  that  FA 
of  lower  forms  of  automation,  like  action  support,  as  opposed  to  higher  LOAs,  like  decision¬ 
making,  resulted  in  superior  performance  even  when  compared  to  full  automation.  Likewise, 
Laois  and  Giannacourou  (1995)  stated  that  automation  tends  to  have  a  more  beneficial  effect 
on  low-level  HIP  functions  in  complex  systems,  like  information  acquisition  and 
communication,  and  less  of  an  effect  on  higher-level  processes,  like  diagnosis,  prediction  and 
decision  making.  Similar  to  Laois  and  Giannacourou  (1995),  Endsley  and  Kaber  (1999) 
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found  that  automation  of  task  planning  and  strategizing  functions  degrades  dynamic  control 
task  performance  compared  to  purely  human  strategizing. 

Level  of  automation  can  also  refer  to  the  amount  of  computer  assistance  provided  to 
an  operator  on  a  particular  system  function.  Parasuraman  et  al.  (2000)  defined  LOA  as  the 
amount  of  computer  assistance  that  is  applied  to  a  particular  information  processing  function, 
varying  from  the  lowest  level  of  automation,  or  manual  control,  to  the  highest  level  of 
automation,  or  full  automation.  They  also  identified  four  different  types  of  automation, 
based  on  Wickens’  (1992)  stages  of  HIP,  including  information  acquisition,  information 
analysis,  decision  selection,  and  action  implementation.  Each  type  of  automation  can  occur 
at  varying  levels. 

This  particular  model  of  automation  has  many  implications  for  ATC.  For  example, 
Parasuraman  et  al.  (2000)  suggest  that  high  levels  of  information  acquisition  and  information 
analysis  should  only  be  incorporated  into  a  system  that  has  been  proven  to  be  very  reliable. 
They  offer  the  converging  runway  display  aid  (CRD A)  as  an  example  of  a  reliable  automated 
ATC  system.  The  CRDA  projects  the  future  paths  of  two  aircraft  landing  on  converging 
runways,  relieving  the  controller  of  having  to  mentally  project  the  aircraft  paths.  Likewise, 
low  levels  of  decision  selection  and  action  implementation  automation  should  be 
incorporated  in  high-risk  functions,  such  as  providing  a  clearance  to  prevent  a  conflict  in 
dense  airspace.  Parasuraman  et  al.  (2000)  suggest  this  situation  should  only  be  automated  at 
a  low  level  so  that  there  is  more  hmnan  involvement  compared  to  the  CRDA. 
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Dynamic  Function  Allocation/Adaptive  Automation 

Hopkin  (1998)  lastly  suggests  that  automation  cannot  only  be  applied  statically  to 
different  types  of  system  functions  at  different  levels,  but  it  can  also  be  dynamically  applied 
to  fimctions  to  switch  control  between  the  human  and  machine.  Kaber  and  Endsley  (1997) 
have  labeled  this  concept  as  dynamic  function  allocation  (DFA),  or  adaptive  automation 
(AA).  Adaptive  automation  uses  one  of  five  strategies  for  scheduling  control  allocations 
between  the  human  and  computers:  (1)  critical  events  —  specific  tactical  events  trigger  control 
allocations  (Scerbo,  1996);  (2)  operator  performance  measurement  -  operator  mental  states 
are  inferred  from  primary  or  secondary-task  performance  and  used  as  a  basis  for  automation 
decisions  (Kaber  &  Riley,  1999);  (3)  operator  physiological  assessment  -  proven  and 
validated  physiological  measures  (e.g.,  heart  rate  variability,  electroencephalogram  signals, 
eye  movements,  etc.)  indicate  workload  and  stress,  and  have  been  used  to  trigger  DFAs 
(Freeman,  Mikulka,  Prinzel,  &  Scerbo,  2000;  Scerbo,  Freeman,  &  Mikulka,  2003);  (4) 
system  performance  modeling  -  a  pre-established  pattern  of  system  behavior  is  used  to 
define  a  schedule  of  manual  and  automated  control  allocations  (Scerbo,  1996);  and  (5)  hybrid 
methods  -  one  or  more  of  the  previously  mentioned  techniques  are  integrated  (Parasuraman 
&  Byrne,  2002). 

The  primary  advantage  of  AA  is  that  automation  can  be  introduced  when  needed, 
reducing  workload,  yet  maintaining  operator  involvement  in  the  control  loop  thereby 
decreasing  the  likelihood  of  the  automation  consequences  discussed  in  Section  1.2.  In  the 
context  of  ATC,  automation  can  be  introduced  during  periods  of  high  workload  (e.g., 
emergency  procedures,  high  traffic  density,  aircraft  non-conformance),  when  the  controller 
most  needs  automation,  and  it  can  be  shut-off  during  low  workload  periods  (e.g.,  low  traffic 
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density)  to  maintain  controller  involvement  in  the  system  (Hilbum  et  al.,  1997;  Parasuraman 
et  al.,  2000).  A  number  of  studies  have  provided  evidence  that  this  type  of  approach  to 
automation  can  be  beneficial  (Kaber  &  Endsley,  1997;  Hilbum  et  al.,  1997;  Clamaim  et  al., 
2002;  Kaber  et  al.,  2002;  Kaber  &  Endsley,  in  press).  Kaber  and  Endsley  (1997)  foimd  that 
AA  produced  significantly  superior  performance  than  completely  manual  or  fully  automated 
control.  Hilbum  et  al.  (1997)  found  similar  results.  They  conducted  research  to  determine 
the  effects  AA  has  on  decision  making  tasks  in  ATC  using  the  previously  described  DA. 

They  developed  three  different  automation  schemes  including  constant  manual  control, 
constant  automation,  and  AA  which  introduced  automation  during  high  air  traffic  simulation 
periods  (critical  event  scheduling  strategy).  They  fovmd  that  the  AA  condition  resulted  in  the 
smallest  increase  in  workload  compared  to  fully  manual  and  automated  control. 

Clamann  et  al.  (2002)  also  conducted  research  to  determine  the  effect  AA  has  on 
HIP  using  forms  of  automation  similar  to  those  presented  by  Parasuraman  et  al.  (2000). 

They  found  that  “humans  are  better  able  to  adapt  to  AA  when  applied  to  lower-order  sensory 
and  psychomotor  functions,  such  as  information  acquisition  and  action  implementation,  as 
compared  to  AA  applied  to  cognitive  (planning  and  decision  making)  tasks”  (Clamann  et  al., 
2002,  p.  346). 

Another  design  parameter  to  address  when  incorporating  AA  into  a  system  is  the 
question  of  vdio  will  decide  when  automation  will  be  invoked,  or  who  has  decision  authority. 
There  are  two  general  forms  of  authority  that  can  be  considered,  computer  mandated  and 
human  mandated.  In  the  former,  a  computer  controller  determines  when  automation  is 
needed.  The  computer  automatically  switches  “on”  machine  control  of  specific  system 
functions.  However,  this  form  of  authority  can  further  remove  the  human  fi'om  the  system 
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control  loop.  The  second  form  of  authority,  human  mandated,  may  involve  the  computer 
suggesting  DFAs  to  the  operator,  and  the  decision  to  switch  to  automation  is  left  up  to  the 
human  (Clamann  &  Kaber,  in  press;  Kaber  &  Endsley,  in  press). 

Adaptive  Automation  is  currently  being  assessed  as  an  option  for  reducing  the 
workload  associated  with  ATC  while  keeping  controllers  involved  in  system  control  loops 
and  maintaining  S  A.  Adaptive  automation  and  other  forms  of  advanced  automation  may  be 
implemented  in  ATC  in  the  future. 

Free  Flight 

Another  approach  to  future  ATC  that  may  incorporate  a  number  of  advanced 
automation  systems  is  free  flight.  Currently,  when  aircraft  fly  under  instrument  flight  rules 
(IFR)  they  establish  a  flight  plan  with  ATC,  which  includes  a  particular  flight  path,  altitude 
and  flight  times  to  fly  to  the  aircraft’s  destination.  If  an  aircraft  needs  to  deviate  from  their 
established  flight  plan  for  any  reason  (e.g.,  weather,  conflict,  emergency  procedures,  etc.) 
they  must  communicate  with  ATC  and  request  a  change  (Nolan,  1999). 

The  FAA  has  proposed  a  complete  restructuring  of  ATC  by  2010  that  will  incorporate 
free  flight,  which  allows  aircraft  to  autonomously  fly  any  heading  or  altitude  during  any 
times  that  are  convenient  for  the  aircraft  (Dillingham,  1998).  If  a  change  is  needed,  pilots 
simply  redirect  their  flight  without  any  communication  with  ATC. 

New  aircraft  displays  being  developed  for  ATC  of  free  flight  present  aircraft  locations 
with  protected  and  alert  zones  which  controllers  can  use  to  safely  monitor  and,  if  needed, 
separate  aircraft  (Nolan,  1999).  Only  if  a  potential  conflict  arises  do  controllers  intervene 
and  actively  resolve  conflicts  (Nolan,  1999;  NRC,  1998).  This  form  of  advanced  automation 
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has  many  benefits,  including  flight  efficiency  and  cost  reductions  due  to  direct,  streamlined 
routes,  and  reduction  in  ATC  personnel  and  equipment  requirements.  However,  fi-ee  flight 
also  presents  many  challenges  associated  with  its  implementation,  primarily  controllers  being 
more  OOTL  during  system  operations. 

1.4  Summary 

Situation  awareness  is  a  prominent  concept  in  human  factors  design  and  development 
of  complex  systems.  It  is  the  perception,  comprehension,  and  projection  of  system  states  in 
an  environment  (Endsley,  1995a).  Situation  awareness  is  critical  for  air  traffic  controllers  to 
maintain  a  mental  picture  of  the  airspace  environment  and  the  aircraft  within  that 
environment  (Niessen  et  al.,  1999).  In  order  for  controllers  to  maintain  appropriate  levels  of 
SA,  spatial,  system,  and  task  awareness  are  all  important  requirements  (Wickens,  2002).  Air 
traffic  control  is  such  a  complex  system,  that  controller  awareness  is  critical  to  identifying 
and  understanding  concurrent  goals  and  tasks  (Endsley  &  Rodgers,  1994),  and  what  the 
temporal  sequence  of  cognitive  activities  is  required  for  successfully  accomplishing  these 
tasks  (Niessen  et  al.,  1999). 

Air  traffic  control  requires  high  levels  of  cognitive  processing,  and  one  approach  for 
alleviating  the  stress  and  workload  among  controllers  is  to  allocate  some  controller  activities 
to  computers,  or  automation  (NRC,  1997;  Parasuraman  &  Riley,  1997).  However,  the 
introduction  of  automation,  though  providing  many  benefits  when  done  properly,  has 
potential  disadvantages.  One  primary  disadvantage  of  introducing  automation  in  ATC  is  the 
possibility  of  removing  the  operator  from  the  control  loop  (Endsley  &  Kaber,  1999).  Forms 
of  traditional  automation  may  cause  such  problems  and  do  not  achieve  the  objective  of 
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human-centered  automation.  Likewise,  by  removing  the  controller  from  the  control  loop, 
there  may  be  adverse  effects  on  SA  (Wickens,  2002;  Endsley,  1996;  Endsley  &  Kiris,  1995; 
Kaber  et  al.,  1999;  Parasuraman  &  Byrne,  2002). 

An  option  for  maintaining  controllers  in  system  control  loops  while  exploiting  the 
benefits  of  automation  is  the  use  of  more  advanced  approaches  to  automation  including 
DFAs.  Such  forms  of  automation  allow  computers  to  accomplish  parts  of  a  task  for  the 
human  at  particular  times  during  task  performance  (Hopkin,  1998;  Kaber  &  Endsley,  1997; 
Wickens  &  Hollands,  2000).  Adaptive  automation  can  allow  for  the  introduction  of 
computer  assistance  to  operators  during  different  phases  of  a  task  when  it  is  most  needed 
(Hopkin,  1998).  As  technological  development  makes  these  forms  of  automation  more 
readily  available,  and  knowledge  of  implementation  methods  is  developed,  AA  and  free 
flight  ATC  systems  may  assume  the  future  role  of  aiding  controllers  to  improve  air  traffic 
efficiency. 


2  Problem  Statement 


As  discussed  in  the  previous  sections,  SA  is  an  important  aspect  of  human-machine 
interactions  in  complex  systems  control.  Numerous  studies  have  defined  methods  for 
measuring  SA  (Endsley,  1995a;  Endsley,  1995c;  Endsley  &  Kaber,  1999;  Hauss  and  Eyferth, 
2003;  Jones  &  Kaber,  in  press).  A  general  result  of  applying  these  methods  to  the  study  of 
human-machine  interaction  is  a  clear  understanding  of  how  system  designs,  and  automation 
in  particular,  can  influence  user  SA.  Endsley  and  Rodgers  (1994)  observed  that  SA  is  critical 
to  ATC  because  of  human  interaction  with  highly  complex  automated  systems.  Numerous 
empirical  studies  have  analyzed  the  impact  of  automation  and  S  A  in  different  contexts  using 
various  measures  (Sarter,  1991;  Carmody  &  Gluckman,  1993;  Endsley,  1996;  Kaber  & 
Endsley,  1997;  Kaber  &  Endsley,  in  press;  Endsley  &  Kaber,  1999;  Kaber,  Onal,  &  Endsley, 
1999). 

Adaptive  automation  has  been  introduced  in  complex  systems  to  prevent  the 
shortcomings  of  conventional  automation,  including  OOTLUF,  but  there  is  limited  evidence 
on  the  extent  to  which  AA  may  succeed  in  moderating  operator  workload  and  facilitating  S A. 
Research  is  needed  to  describe  the  SA  effect  of  AA  in  the  context  of  advanced  ATC  systems. 
This  study  assessed  the  performance,  SA  and  workload  effects  of  AA  of  four  different  stages 
of  HIP  in  an  ATC  simulation.  The  following  objectives  were  established  at  the  onset  of  the 
research: 

(1)  define  a  measure  of  S A  in  an  ATC  simulation  that  is  a  sensitive  and  reliable 
indicator  of  automation  state  changes  as  part  of  AA; 
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(2)  empirically  assess  the  S  A  measure  for  use  in  investigating  the  effectiveness  of  AA 
of  various  ATC  information  processing  functions;  and 

(3)  determine  the  influence  of  AA  in  an  ATC  simulation  on  operator  SA. 

The  results  of  this  study  were  expected  to  be  used  as  a  basis  for  developing  a  real¬ 
time  measure  of  SA  in  ATC,  which  could  be  used  as  a  trigger  for  DFAs  in  complex  systems 
control  in  order  to  manage  operator  workload  and  potentially  prevent  OOTL  performance 
problems. 
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3  Methods 


The  experiment  conducted  as  part  of  this  research  involved  human  subject  trials  to 
measure  performance,  workload,  and  SA  in  a  simulated  ATC  task  under  various  modes  of 
AA.  The  latest  version  of  an  ATC  simulator.  Multitask©  (version  3.0),  developed  and  used 
by  Endsley  and  Kaber  (1999),  Kaber  et  al.  (2002)  and  Clamann  et  al.  (2002),  was  used  in 
conjunction  with  a  gauge-monitoring  task  as  the  research  test-bed. 

3.1  Tasks 

3.1.1  Multitask 

Multitask©  is  a  low  fidelity  ATC  approach  control  simulator.  The  objectives  of  an 

operator  are  to  contact  aircraft  appearing  on  a  radar  display,  make  any  necessary  changes  to 

pre-existing  aircraft  clearances  based  on  their  potential  to  cause  a  conflict,  and  safely  land 

aircraft  at  one  of  two  airports.  One  of  four  modes  of  automation  (of  various  HIP  functions  as 

part  of  the  task),  and  manual  control,  can  be  used  in  the  simulation.  Task  performance  is 

based  on  the  number  of  aircraft  safely  landed  at  an  airport,  the  number  of  aircraft  that 

encounter  conflicts,  and  the  number  of  clearance  amendments  administered. 

The  simulator  display  includes  a  menu  bar,  radarscope,  and  control  and  status  boxes 

(see  Figure  3.1).  The  menu  bar  is  used  to  select  experimental  trial  settings,  skill  levels,  and 

automation  modes,  all  of  which  will  be  discussed  in  later  sections.  The  radarscope  represents 

approximately  15,000  square  nautical  miles  (nm)  of  airspace  within  the  users  control,  and  is 

separated  into  four  quadrants  by  horizontal  and  vertical  lines,  each  delineating  the  north, 

south,  east,  and  west  cardinal  directions  on  the  display.  Concentric  circles  are  spaced  on  the 
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display,  each  representing  10  nm  increments  from  the  display  center.  Near  the  center  of  the 
radarscope  are  two  airports,  one  10  nm  west  of  the  center  and  another  10  nm  east  of  the 
center  (with  20  nm  between  the  airports).  Each  airport  has  two  runways.  Eight  equally 
spaced  holding  fixes,  represented  on  the  display  by  small  gray  circles,  are  located  30  nm 
from  the  center  of  the  radarscope. 


Figure  3.1  Multitask  display  in  manual  mode 


Each  simulated  aircraft  is  represented  on  the  display  by  a  triangle  icon  and  a  data  tag 
depicting  the  aircraft’s  call  sign  (see  Figure  3.2).  The  aircraft  data  tag  is  similar  to  those 
found  on  ATC  radar  displays  (NRC,  1997).  The  icons  first  appear  at  the  perimeter  of  the 
display  on  one  of  eight  approach  trajectories  and  move  toward  one  of  the  two  airports, 
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destined  for  one  of  the  runways.  An  aircraft’s  starting  location,  airport,  and  runway  are  all 
randomly  assigned,  and  the  aircraft  appear  one  at  a  time  with  a  1-20  s  buffer  between  aircraft. 


_ — 

A  MC2864 

Figure  3.2  Aircraft  icon 

When  an  aircraft  appears  on  the  radarscope,  it  is  presented  as  a  flashing  white 
triangle,  indicating  that  it  has  not  yet  been  contacted.  Once  an  aircraft  is  contacted  (selected 
and  queried  using  a  mouse  controller),  the  icon  becomes  a  solid  white  triangle,  and  the 
aircraft’s  predetermined  clearance  (including  call  sign,  aircraft  speed,  destination  airport,  and 
destination  runway)  is  displayed  in  a  data  box  on  the  left  side  of  the  Multitask©  display  (see 
top  of  Figure  3.3).  If  an  aircraft  is  placed  in  a  holding  pattern,  the  icon  becomes  yellow  until 
the  aircraft  is  advised  to  resume  its  approach  to  the  destination  airport,  at  which  time  it 
becomes  white  again. 

The  aircraft  icons  represent  one  of  three  possible  types  of  aircraft;  commercial, 
private,  or  military.  The  aircraft  call  sign  is  represented  by  an  alphanumeric  designator, 
including  an  aircraft  type  letter  designator  and  a  numerical  designator  as  shown  in  Table  3.1. 


Table  3.1  Aircraft  designators 


Aircraft  Type 

Letter  designator 

Four  digit  designator 

Military 

M 

Sequential  number 

Commercial 

Random  2-character  airline  designator 

Random  number 

Private 

GA 

Sequential  number 
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The  three  aircraft  types  occur  in  the  simulation  with  different  frequencies  (see  Table 
3.2).  The  type  of  aircraft  also  dictates  the  possible  range  of  speed  for  the  vehicle.  The  exact 
speed  is  determined  based  on  a  standard  amount  of  time  required  for  each  aircraft  type  to 
reach  its  destination  (without  a  controller  clearance  amendment).  The  average  time  and 
speed  values  (real-time)  for  each  type  of  aircraft  are  shown  in  Table  3.2.  The  destination 
airports  include  A1  for  the  west  airport  or  A2  for  the  east  airport,  and  the  runways  include  R1 
or  R2. 


Table  3.2  Aircraft  type  parameters 


Aircraft  Type 

Frequency 

Travel  time 

Average  speed 

Military 

10% 

21  minutes 

200  kts 

Commercial 

70% 

25  minutes 

170kts 

Private 

20% 

42  minutes 

100  kts 

The  control  box  allows  the  user  to  communicate  with  all  of  the  aircraft  currently  on 
the  radarscope  and  to  request  changes  to  aircraft  parameters.  It  includes  the  previously 
mentioned  data  box,  eight  command  buttons,  and  a  history  box  (see  Figure  3.3). 


SJ6D96  S168  A2  R1  J 

i  Quev  1 

Reduce 

Speed 

j 

Resume 

Change 

Airpoft 

Change 

Ruhw^ 

Figure  3.3  Control  box 
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The  data  box  is  located  on  the  top  of  the  control  box.  The  information  in  this  box  is 
displayed  imtil  another  aircraft  is  contacted.  The  eight  control  buttons  facilitate  six  clearance 
change  commands,  including  query,  reduce  speed,  hold,  resume,  change  airport,  and  change 
runway.  The  two  action  commands  include  submit  and  cancel.  The  ‘Query’  command  is 
used  to  initiate  communication  with  an  aircraft  and  obtain  the  aircraft’s  flight  identifier  and 
parameters.  The  ‘Reduce  Speed’  command  simply  reduces  the  aircraft’s  speed  by  a  set 
amount,  and  the  original  speed  cannot  be  subsequently  reassigned.  The  ‘Hold’  command  is 
used  to  request  that  an  aircraft  fly  directly  to  the  nearest  of  eight  holding  fixes  and  remain 
there  for  either  30  min.  or  until  advised  to  continue  to  their  assigned  airport  and  runway  (the 
‘Resume’  command).  The  ‘Change  Airport’  and  ‘Change  Runway’  commands  advise 
aircraft  to  change  from  their  randomly  assigned  clearance  to  the  other  of  the  two  possible 
destinations.  The  ‘Submit’  button  must  be  used  after  selecting  a  desired  clearance  command, 
and  ‘Cancel’  may  be  used  to  prevent  an  aircraft  from  further  processing  a  command.  All  of 
the  command  keys  are  used  to  prevent  possible  conflicts  between  aircraft  while  maintflining 
efficiency  (e.g.,  issuing  the  fewest  number  of  clearance  revisions).  The  history  box,  located 
below  the  command  buttons  (see  Figure  3.3),  displays  the  communications  with  the  aircraft 
in  text  form,  simulating  actual  ATC  communications  (e.g.,  GA4:  Change  airport). 

Beneath  the  command  box  is  an  automation  aid  display  box  (see  left-center  of  Figure 
3.4).  This  box  contains  information  pertinent  to  the  current  level  of  automation,  and  is 
inactive  under  the  manual  control  (no  automation)  condition  and  information  acquisition 
modes.  The  information  in  this  box  is  described  below  for  each  mode  of  Multitask© 
automation. 
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Figure  3.4  Decision  making  automation  display  screen 
An  automation  status  control  box  is  located  below  the  automation  aid  box  and 
indicates  whether  automation  or  fully  manual  control  is  currently  being  utilized  (see  left- 
bottom  of  Figure  3.4).  During  AA  trials  when  the  computer  intervenes  in  human  operator 
control  (mandated  automation),  the  automation  status  control  box  displays,  “Automation  is 
ON,”  and  during  manual  control  portions  of  AA  trials  (computer  return  of  control  to  the 
human)  the  box  displays,  “Automation  is  OFF.” 

Multitask©  is  capable  of  operating  imder  one  of  the  following  five  modes  of 
automation  during  a  simulation  trial; 

1 .  Manual  control  -  No  additional  assistance  is  provided.  Operations  are  performed 
as  described  in  the  previous  sections  (see  Figure  3.1). 
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2.  Information  acquisition  -  A  scan  line  rotates  clockwise  around  the  radar  display, 
and  as  it  passes  over  an  aircraft  icon,  a  Trajectory  Projection  Aid  (TP A)  for  that 
aircraft  is  presented  for  2  s.  The  TPA  shows  the  aircraft  destination  and  route  in 
the  form  of  a  line  connecting  the  aircraft  and  the  airport  or  holding  fix.  The 
aircraft  speed  (in  knots),  destination  airport,  and  destination  runway  are  affixed  to 
the  center  of  the  TPA  (see  upper-left  radarscope  quadrant  of  Figure  3.5).  This 
mode  of  automation  allocates  the  information  gathering/data  collection  stage  of 
HIP  to  machine  control  (Wickens,  1992).  The  ARTS  and  the  TMA  incorporate 
automation  similar  to  this  by  including  data  blocks  and  graphic  displays  of  aircraft 
tracks  on  radar  displays  (Debelack,  Dehn,  Muchinsky,  &  Smith,  1995;  Nolan, 
1999). 


P  recessing,  ,  i  .  , - 

■  .  .  C'iiKV  -  .1  -iv:::- 

i 

.  !  ■  [. 

_  ..  .--A  i./.: 

.  -I  .  ■  £;hi':i£;  ■  T;;;;-. 

/■' 

.J-  ™  ^  --- . 

IlmSIMl;  CIWMing  rur^vf 

,  /  ,  ,  'n  ft!  fu  ^ 

'  ;  '  r'  ^  "  L  -  '■  I  i  i 

;  C;  ■. 

*  \  \ 

’o  ..--o' 

"‘■■■A 

i  J AllPi- rdl:  -  OH  ” 

■J  -l..  - ■■  - 

1 

I 

I  : 

! 

. n- . 

Figure  3.5  Information  acquisition  automation  display  screen 
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3.  Information  analysis  -  Information  pertinent  to  each  of  the  contacted  aircraft  on 
the  radarscope  is  displayed  in  the  automaton  aid  box  including  the  aircraft’s  call 
sign,  destination  airport,  destination  runway,  speed,  and  distance  (nm)  fi-om  the 
destination  airport  (see  Figure  3.6).  A  final  column  in  the  table,  ‘Conf,’  denotes 
the  call  sign  of  any  aircraft  that  are  currently  in  conflict  with  each  other.  This 
mode  of  automation  may  reduce  operator  workload  associated  with  the  integration 
of  perceived  information  with  long-term  memories  in  working  memory  as  part  of 
HIP  (Wickens,  1992).  This  mode  of  automation  is  very  similar  to  the  current 
User  Request  Evaluation  Tool  (URET)  and  the  Prediction/Resolution  Advisory 
Tool  (PRAT)  used  by  air  traffic  controllers,  which  alert  controllers  to  potential 
conflict  situations  (Nolan,  1999). 
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Figure  3.6  Information  analysis  automation  aid  box 
4.  Decision  making  —  In  addition  to  conflict  alerting,  recommendations  for  conflict 
resolution  are  provided  in  the  automation  aid  box.  Information  on  conflicting 
aircraft,  the  recommended  clearance  change  (speed,  hold,  resumption,  airport,  or 
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no  option),  and  which  aircraft  to  advise  of  a  necessary  change  are  all  displayed 
together  (see  Figures  3.4  and  3.7).  Up  to  ftiree  automated  clearance 
recommendations  are  displayed  in  the  automation  aid  box  at  a  time  and  they  are 
listed  in  order  of  priority.  Runway  changes  are  not  recommended  by  the 


automation.  This  is  an  intended  limitation  in  the  automation  in  comparison  to 
human  manual  control  to  ensure  active  operator  involvement  in  the  simulation. 
This  mode  of  automation  assumes  responsibility  for  the  HIP  requirements 
associated  with  the  decision  and  response  selection  aspects  of  the  task  (Wickens, 
1992).  It  is  similar  to  the  conflict-resolution  capability  of  the  PRAT  (Nolan, 


1999). 
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Figure  3.7  Decision  making  automation  aid  box 


5.  Action  implementation  -  This  mode  of  automation  simulates  the  hand-off  of  air 
traffic  control  from  approach  control  to  local-tower  control,  and  the  tower 
automatically  maintains  full  control  and  responsibility  of  aircraft  within  20  nm  of 
the  center  of  the  radarscope.  Automation  of  clearances  for  aircraft  within  this 
area  prevents  any  conflicts  (as  described  in  the  next  section)  after  hand-off  to 
tower  control.  The  action  implementation  display  includes  a  decision  aid  which 
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simply  summarizes  the  classification  and  number  of  aircraft  on  the  display  (see 
Figure  3.8).  The  aid  is  included  as  part  of  this  mode,  in  part,  to  facilitate 
consistency  in  the  interface  information  content  across  conditions.  Action 
implementation  automation  assumes  the  HIP  requirement  of  response  execution 
as  part  of  the  ATC  simulation  (Wickens,  1992).  Proposed  data  link  technology  is 
currently  being  explored  as  a  replacement  for  voice  communication  between 
controllers  and  aircraft  (Debelack  et  al.,  1995;  Dillingham,  1998).  One  feature  of 
this  mode  of  automation,  similar  to  the  Multitask©  action  implementation  mode 
of  automation,  is  the  computer  automatically  hands-off  aircraft  fi'om  one  control 
sector  to  another  (Nolan,  1999;  NRC,  1997). 
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Figure  3.8  Action  implementation  automation  aid  box 

Each  mode  of  automation  is  a  unique  mode  of  computer  assistance  simulating  only 
one  HIP  function.  That  is,  the  modes  of  automation  are  not  built-up  from  one  mode  to  the 
next.  For  example,  the  information  analysis  mode  of  automation  only  provides  computer 
assistance  of  the  information  analysis  aspect  of  the  task,  not  information  analysis  and 
information  acquisition. 
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Multitask©  perfonnaiice  is  measured  in  terms  of  the  number  of  aircraft  cleared, 
aircraft  conflicts,  aircraft  collisions,  and  the  number  of  clearance  amendments  administered. 
This  data  is  recorded  during  simulation  trials  and  displayed  at  the  end  of  the  task  (see  Figure 
3.9).  Aircraft  arriving  safely  at  an  airport  are  considered  cleared  aircraft.  Aircraft  traveling 
within  3  nm  of  another  aircraft,  as  they  travel  towards  their  assigned  airport,  or  aircraft  that 
are  within  20  nm  of  the  center  of  the  radarscope  (second  concentric  circle)  and  destined  for 
the  same  runway  at  the  same  airport,  are  considered  conflicts.  Aircraft  that  simultaneously 
arrive  at  the  same  airport  destined  for  the  same  runway,  or  aircraft  that  come  in  contact  with 
each  other  constitute  collisions,  in  which  case  auditory  feedback  is  provided  and  both  aircraft 
are  removed  from  the  screen.  In  addition,  the  number  of  clearance  changes,  and  successful 
and  unsuccessful  (nonconformance)  changes  are  recorded  as  measures  of  performance. 
Unsuccessful,  or  nonconformance,  changes  result  from  either  an  erroneous  controller  request 
or  a  random,  infrequent  aircraft  noncompliance  simulating  real  ATC  communication.  In 
general,  operators  attempt  to  minimize  the  number  of  necessary  clearance  changes.  (More 
details  on  the  simulation  performance  measures  are  presented  in  the  description  of  the 
independent  and  dependent  variables  of  the  experiment  in  Section  3.5.) 
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Test  complete. 

Aircraft  Landed:  9 
Potential  Collisions:  2 
Actual  Collisions:  0 

Aircraft  Slowed:  3 
Holding  Patterns:  1 
Redirects:  3 
Runway  Changes:  2 
Sews:  13 


Figure  3.9  Multitask©  performance  measures 

The  experimenter  controls  the  mode  of  automation  applied  to  the  simulation,  the 
duration  of  trials,  and  the  skill  level.  The  skill  level  is  defined  by  the  number  of  aircraft 
(from  3  to  9)  that  are  present  on  the  radarscope  at  any  given  time.  During  all  training  and  test 
trials  as  part  of  this  experiment,  7  aircraft  appeared  on  the  radarscope  from  moment-to- 
moment. 


3.1.2  Gauge-monitoring  Task 

The  gauge-monitoring  task  used  in  this  research  was  similar  to  the  secondary  task 

developed  by  Kaber  and  Riley  (1999)  and  used  by  Clamann  et  al.  (2002)  and  Kaber  et  al. 

(2002).  It  was  a  simple  loading  task  which  also  contributed  to  the  realism  of  the 

experimental  test  scenario,  as  human  operators  of  complex  automated  systems  are  often 

required  to  perform  multiple  tasks  simultaneously  creating  high  workload  conditions.  It 

included  a  fixed  scale,  moving  pointer  display  with  a  central  “acceptable”  range  bordered  on 

either  side  by  two  “unacceptable”  ranges  (see  Figure  3.10).  The  acceptable  range  was 

colored  green,  the  unacceptable  ranges  were  colored  red,  and  two  small  transitional  areas 
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were  colored  blue.  The  transitional  ranges  were  also  considered  “acceptable”  gauge  values. 
The  user’s  goal  was  to  prevent  the  randomly  drifting  pointer  from  moving  into  either 
“imacceptable”  range  by  maintaining  it  in  the  “acceptable”  range  using  particular  keys  on  a 
keyboard  (‘Shift’  to  move  the  pointer  up  or  ‘Ctrl’  to  move  the  pointer  down)  depending  on 
which  direction  the  pointer  drifted.  If  the  pointer  drifted  into  an  “unacceptable”  range, 
depressing  the  appropriate  key  returned  the  pointer  to  the  center  of  the  “acceptable”  range. 
After  returning  the  pointer  to  the  “acceptable”  range,  it  continued  to  drift  randomly  xmtil  the 
end  of  a  trial.  The  pointer  did  not  dwell  in  an  unacceptable  region  for  more  than  3  s  at  a 
time.  Three  seconds  was  determined  by  Clamann  et  al.  (2002)  as  a  sufficient  amount  of  time 
for  subjects  to  register  a  deviation  while  also  performing  the  primary,  ATC  simulation.  The 
design  of  the  gauge  task  allowed  for  approximately  six  pointer  deviations,  or  signals,  to  occur 
per  minute  with  a  variance  of  plus-or-minus  one  signal. 


Figure  3.10  Gauge-monitoring  display 

Errors  in  the  gauge-monitoring  task  were  defined  as  the  pointer  deviating  into  either 
of  the  “unacceptable”  ranges  (a  signal)  without  a  control  command  from  the  operator  to 
correct  the  deviation  (a  miss).  Errors  were  also  recorded  when  operators  commanded  an 
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unnecessary  correction  (a  false  alarm',  e.g.,  a  ‘Ctrl’  key  press  when  the  pointer  was  in  the 
green  or  blue  ranges). 

The  gauge  task  provided  an  index  of  operator  workload  in  the  Multitask©  simulation. 
A  low  score  in  monitoring  implied  a  high  level  of  workload  in  the  ATC  simulation  and  vice 
versa.  The  task  also  served  as  a  basis  for  invoking  changes  in  the  allocation  of  Multitask© 
functions  from  manual  control  to  one  of  the  modes  of  automated  control.  When  secondary- 
task  (gauge)  performance  was  good,  suggesting  a  reduction  in  operator  workload,  Multitask© 
returned  to  completely  manual  control. 

In  general,  gauge  task  performance  was  recorded  as  a  hit-to-signal  ratio,  or  the 
number  of  times  a  subject  detected  the  pointer  in  an  “unacceptable”  range,  divided  by  the 
number  of  signals.  Upper  and  lower  performance  bounds  for  gauge-monitoring  were 
determined  during  subject  training  as  part  of  the  experiment  for  each  mode  of  automation. 
Pairs  of  subjects  performed  gauge-monitoring  practice  trials  under  a  randomly  assigned 
mode  of  Multitask©  automation,  and  dieir  performance  was  used  to  establish  primary  task 
“overload”  conditions  for  all  subjects  exposed  to  the  same  mode  of  automation  during  the 
experimental  test  trials.  Pairs  of  subjects  performed  gauge  practice  trials  for  the  information 
acquisition,  information  analysis,  decision  making,  and  action  implementation  modes  of 
Multitask©  automation.  The  upper  and  lower  performance  boimds  were  used  as  a  basis  for 
determining  when  DFAs  should  occur  in  the  Multitask©  simulation  under  the  various  modes 
of  automation;  that  is,  the  AA  was  implemented  based  on  operator  secondary-task 
performance.  These  procedures  are  explained  in  more  detail  in  Section  3.6. 
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3.2  Participants 

Eight  subjects  were  recruited  to  participate  in  this  experiment  from  the  North 
Carolina  State  University  student  population.  Subjects  ranged  in  age  from  23  to  27  years. 

All  subjects  had  20/20,  or  corrected  to  normal,  vision  and  personal  computer  experience. 
None  of  the  subjects  had  flying  or  ATC  experience.  Subjects  were  compensated  at  a  rate  of 
$10  per  hour  for  their  participation.  Subject  were  informed  of  an  additional  incentive  at  the 
beginning  of  the  experiment,  a  $30  gift  certificate,  which  was  awarded  to  the  top  performer. 

3.3  Equipment  and  Environment 

The  equipment  used  in  the  experiment  included  a  high-performance  graphics 
workstation  with  512  MB  of  RAM  and  dual  2  GHz  Pentium  Xeon  processors.  Multitask© 
and  the  gauge-monitoring  task  were  presented  on  two  17-inch,  digital  flat  panel  displays  (see 
Figure  3.11).  Two  keyboards  and  two,  2-D  computer  mice  were  integrated  with  the 
computer  system.  Both  the  subject  and  the  experimenter  used  a  keyboard  and  mouse  during 
test  trials.  The  subject’s  keyboard  and  mouse  were  used  to  control  the  Multitask©  and 
gauge-monitoring  tasks,  while  the  experimenter’s  interface  controls  were  used  to  facilitate 
simulation  trial  freezes  (second  set  of  controls  not  shown  in  Figure  3.1 1). 
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Figure  3.11  Workstation 

All  trials  took  place  in  a  small  office  with  no  windows  and  the  overhead  lights  turned 
off  to  prevent  glare  on  the  computer  screens.  A  small  lamp  was  set  up  behind  the  computer 
monitors.  A  separate  desk  and  computer  were  set  up  adjacent  to  the  simulation  workstation 
and  were  used  by  subjects  to  complete  SA  questionnaires  (described  in  Section  3.5).  The 
experimenter  sat  out  of  view  of  subjects  and  took  particular  care  to  not  distract  subjects  while 
the  simulation  was  running.  Simulation  freezes  were  periodically  conducted  to  allow 
subjects  to  turn  and  respond  to  the  SA  questions  at  the  second  computer  workstation  while 
the  experimenter  obtained  correct  answers  to  the  SA  queries  from  the  Multitask©  console. 
The  specific  experimental  procedures  are  discussed  in  detail  in  the  following  sections. 
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3.4  Experimental  Design 

The  experiment  followed  a  completely  within-subjects  design  in  which  all  eight 
subjects  completed  two,  30-min.  trials  imder  each  of  the  five  modes  of  Multitask© 
automation  (i.e.,  the  design  was  replicated  to  allow  for  estimation  of  experimental  error). 

Each  experiment  trial  consisted  of  both  manual  and  automated  minutes  (with  the  exception  of 
manual  control  condition  trials  which  involved  only  manual  minutes),  depending  on  the 
DFAs  that  occurred  during  the  ATC  simulation  on  the  basis  of  subject  workload  (as  indicated 
by  gauge-monitoring  performance).  Consequently,  all  performance,  workload  and  SA 
response  measures  were  calculated  for  both  automated  and  manual  control  periods  as  part  of 
AA  trials.  The  experiment  yielded  10  manual  and  8  automated  performance  scores  for  each 
subject.  In  total,  eighty  composite  performance  observations  (manual  and  automated 
minutes)  were  gathered  across  subjects  (8  subjects  x  2  trials  x  5  control  modes),  and  64 
manual  and  automated  performance  observations  were  collected  (8  subjects  x  2  trials  x  4 
control  modes)  on  AA  conditions  (excluding  the  manual  control  condition  trials). 

3.5  Variables 

The  independent  variable  (IV)  manipulated  in  this  study  was  the  mode  of  AA  applied 
to  the  Multitask©  simulation,  including  completely  manual  control,  information  acquisition 
automation,  information  analysis  automation,  decision  making  automation,  and  action 
implementation  automation.  Under  the  AA  conditions,  each  mode  of  automation  switched 
“on”  and  “off’  depending  upon  operator  workload  states. 

The  dependent  variables  included  Multitask©  performance,  which  was  recorded  in 
terms  of  cleared  aircraft,  conflicts,  and  collisions.  Another  dependent  variable  was  gauge- 
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monitoring  task  performance,  recorded  as  a  hit-to-signal  ratio.  Both  of  these  measures  were 
captured  during  each  minute  of  a  test  trial. 

Percent  correct  responses  to  SA  queries  were  also  recorded  during  the  experiment 
using  the  Situation  Awareness  Global  Assessment  Technique  (SAGAT),  developed  and 
validated  by  Endsley  (1995a;  1995c).  The  SAGAT  is  a  direct,  objective  measure  of  SA. 

One  approach  to  developing  SA  queries  as  part  of  the  SAGAT  methodology  is  to  conduct  a 
GDTA  on  the  simulation  being  used.  A  portion  of  a  GDTA  developed  for  the  Multitask© 
subgoal  “acquire  aircraft  information”  is  presented  here  (a  GDTA  for  Multitask©  is 
presented  in  its  entirety  in  Appendix  A): 

1.  acquire  aircraft  information 
1.1  locate  aircraft 

in  what  display  sector  is  the  aircraft  located? 
aircraft  position 

how  many  other  aircraft  are  located  in  that  sector? 
aircraft  position 
location  of  other  aircraft 

In  the  GDTA  segment  shown  above,  the  goals  and  subgoals  were  broken-down  to 
identify  major  decisions  that  must  be  made  to  accomplish  the  goals:  “in  what  display  sector 
is  the  aircraft  located?”  and  “how  many  other  aircraft  are  located  in  that  sector?”  These 
decisions  led  to  identification  of  the  perception,  comprehension,  and  projection  requirements 
for  accomplishing  goals  in  the  Multitask©  domain.  These  questions  and  S  A  requirements 
were  used  to  develop  SA  queries  (see  Appendix  B).  Examples  of  a  Level  1, 2,  and  3 
question  used  in  this  study  are  presented  here: 
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Table  3.3  Level  1  example  SAGAT  question 


Aircraft  # 

What  is  the 
aircraft's  call 
sign? 

1 

2 

3 

4 

5 

6 

7 

Criterion: 

none 

Table  3.4  Level  2  example  SAGAT  question 


Aircraft  # 

What  is  the 
aircraft's 
distance  to 
destination? 
(nm) 

1 

2 

3 

4 

5 

6 

7 

Criterion: 

within  5  nm 

Table  3.5  Level  3  example  SAGAT  question 


Aircraft  # 

When  will  the 
aircraft  arrive 
at  the 
destination 
airport?  (min 
from  now) 

1 

2 

3 

4 

5 

6 

7 

Criterion: 

within  2  min 
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Endsley’s  (1995c)  implementation  of  the  SAGAT  involves  administering  SA  queries 
to  operators  during  simulation  trials.  The  SAGAT  procedure  requires  the  simulation  to  be 
frozen  one  or  more  times  during  a  trial.  Typically,  the  subject  is  presented  with  a  map  of  the 
work  environment  and  asked  to  locate  various  elements  of  the  scenario  (Jones  &  Kaber,  in 
press).  In  this  experiment,  a  radarscope  display  graphic  was  presented  to  subjects,  and  the 
subjects  identified  the  current  locations  of  the  7  aircraft  on  the  display  (see  Appendix  B). 
This  graphic,  or  key,  forms  the  basis  for  questions  administered  during  the  freeze.  As  the 
subject  answers  questions  about  specific  aspects  of  the  task,  a  computer  or  experimenter 
records  the  “ground  truth”  of  the  simulation  scenario.  Subsequent  to  responding  to  the 
questions,  the  simulation  is  resumed  for  the  subject. 

Endsley  (1995c)  identified  several  guidelines  for  implementing  SAGAT  including: 

•  The  timing  of  the  freezes  should  be  random  in  order  to  prevent  subjects  from 
anticipating  a  query  and  to  ensure  queries  do  not  occur  during  the  same  phases  of  a 
simulation  or  that  they  only  occur  during  high  activity. 

•  The  queries  should  not  occur  within  the  first  3-5  minutes  of  a  trial  to  allow  a  subject 
time  to  develop  SA. 

•  Two  queries  should  not  occur  within  1  min.  of  each  other  to  allow  subjects  time  to 
reacquire  SA  after  a  freeze. 

•  Multiple  freezes  during  a  trial  are  appropriate.  Endsley  (1995c)  found  that  3  stops 
during  a  15-min  trial  did  not  adversely  affect  subject  performance  in  a  simulation. 

•  An  entire  experiment  should  collect  30-60  SA  samplings  per  experimental  condition. 
Responses  to  SAGAT  queries  are  scored  as  either  correct  or  incorrect.  Jones  and  Kaber  (in 

press)  said  the  SAGAT  data  provides  diagnostic  information  on  an  individual  query  basis, 
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and  the  scores  across  queries  are  generally  not  aggregated.  The  responses  to  each  query  are 
typically  analyzed  separately;  that  is,  all  of  die  responses  to  one  query  are  analyzed  across 
conditions.  Many  researchers  have  effectively  used  this  approach  to  SAGAT  data  analysis 
including  the  evaluation  of  pilot  SA  with  aircraft  cockpit  systems  (Endsley,  1995b), 
controller  SA  in  ATC  (Endsley  &  Kiris,  1995),  and  radar  monitors  in  the  theater  of  defense 
tasks  (Bolstad  &  Endsley,  2000).  Other  recent  research  on  SA  in  ATC  (Hauss  &  Eyferth, 
2003)  has  also  demonstrated  that  analyzing  SA  questions  individually  can  be  an  effective 
method  on  assessing  controller  SA.  The  results  for  each  query  are  then  generalized  to  the 
appropriate  SA  Level,  and  the  scores  for  each  experimental  condition  serve  as  a  direct 
indicator  of  controller  SA  during  those  conditions.  Jones  and  Kaber  (in  press)  also  describe  a 
method  of  analysis  in  which  SAGAT  queries  are  categorized  according  to  the  levels  of  SA 
defined  by  Endsley  (1995a)  and  composite  scores  for  Level  1, 2,  and  3  SA  are  computed 
based  on  the  accuracy  of  subject  responses  to  the  sets  of  questions.  This  was  the  primary 
method  of  SA  data  analysis  planned  for  this  study.  Statistical  analyses  of  the  SA  scores  in 
this  study  is  discussed  in  more  detail  in  Section  4. 

The  key  advantage  of  the  SAGAT  is  that  it  provides  diagnostic  information  about 
specific  elements  of  operator  S  A,  which  can  be  collected  during  a  trial  without  increasing 
operator  workload  (Endsley,  1996;  Jones  &  Kaber,  in  press).  A  disadvantage  of  SAGAT  is 
its  potential  intrusiveness  in  operator  performance;  however,  some  studies  have  found  that 
SAGAT  stops  are  not  disruptive  to  subsequent  operator  simulation  behavior  (Endsley, 

1995c).  The  specific  implementation  of  the  SAGAT  methodology  in  this  experiment  is 
described  in  the  next  section. 
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3.6  Procedures 


Each  subject  participated  in  a  simulation  training  and  practice  session,  and  10  test 
trials.  The  overall  schedule  for  the  experiment  is  presented  in  Table  3.6.  The  procedures 
began  with  an  introduction  and  equipment  familiarization  period.  During  this  time,  the 
subjects  completed  a  demographie  survey  (see  Appendix  C)  and  their  responses  were 
included  in  a  database  for  the  experiment.  An  informed  consent  form,  participation  payment 
form,  and  contest  rules  and  score  sheet  were  also  reviewed  and  signed  by  subjects  during  this 
period  (see  Appendices  D,  E,  and  F).  It  is  important  to  note  that  the  experiment  duration 
presented  in  the  informed  consent  form  varies  slightly  from  the  final  experiment  schedule  (10 
and  13  hrs,  respeetively)  due  to  the  experiment  lasting  longer  than  expected.  During  the 
introduction  and  familiarization  period,  the  subjects  were  also  shown  the  computer  system 
monitors,  keyboard,  and  mouse  to  be  used  during  the  simulation. 


Table  3.6  Experiment  timetable 


Day  1 

Introduction  and  equipment  familiarization 

15 

Multitask©  training  imder  manual  control 

15 

Gauge-monitoring  training 

5 

Information  acquisition  automation  training 

15 

Information  analysis  automation  training 

15 

Decision  making  automation  training 

15 

Action  implementation  automation  training 

15 

Break 

Adaptive  automation  training 

15 

Dual-task  (Multitask©  and  gauge-monitoring)  practice* 

25 

SAGAT  familiarization 

10 

Dual-task  and  SAGAT  practice* 

50 

Total: 

205 

55 


Table  3.6  Experiment  timetable  (continued) 


Day  2 

Simulation  Review 

15 

Experimental  trial  1** 

Experimental  trial  2** 

Break 

10 

Experimental  trial  3** 

50 

Experimental  trial  4** 

50 

Break 

10 

Experimental  trial  5** 

50 

Total: 

285 

Day  3 

Simulation  Review 

15 

Experimental  trial  6** 

■bhhesi 

Break 

■■■^^■001 

Experimental  trial  7** 

50 

Experimental  trial  8** 

Break 

10 

Experimental  trial  9** 

Experimental  trial  10** 

50 

Debriefing 

10 

Total: 

295 

Total  duration  for  both  days: 

*  Subjects  were  randomly  assigned  to  modes  of  automation  with  equal  numbers  of 
subjects  (n=2)  experiencing  each  mode 

**  The  order  in  which  subjects  were  tested  under  the  various  modes  of  automation  was 
completely  randomized 

Following  the  introduction  and  equipment  familiarization  period,  a  training  and 
practice  session  was  conducted.  First,  Multitask©  and  gauge-monitoring  training  was 
conducted.  Then,  participants  underwent  training  sessions  on  each  of  the  4  modes  of 
automation  and  the  AA  function  of  Multitask©.  This  training  included  descriptions  of  the 
modes,  screen  familiarization,  and  a  short  practice  trial  for  each  mode  of  automation.  The 
simulation  training  was  followed  by  a  dual-task  practice  trial  in  which  both  the  Multitask© 
and  gauge-monitoring  task  were  performed  under  a  randomly  assigned  mode  of  Multitask© 
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automation.  Subsequently,  subjects  were  familiarized  with  the  SAGAT,  and  sample 
questions  were  presented.  Finally,  a  dual-task  practice  session,  involving  performance  under 
the  same  randomly  assigned  mode  of  automation  was  conducted,  with  3  SAGAT  simulation 
freezes,  each  requiring  subjects  to  respond  to  6  SA  questions.  The  SAGAT  questions 
presented  during  the  practice  session  were  randomly  selected  from  the  questions  to  be  posed 
during  the  actual  experimental  trials.  Each  subject  performed  the  dual-task  plus  SAGAT 
practice  trial  imder  one  mode  of  automation  (see  Table  3.7  for  schedule  of  training  and 
testing  trials  on  a  per  subject  basis),  and  the  average  gauge-monitoring  task  performance  and 
standard  deviation  (SD)  were  recorded.  These  measures  were  used  during  the  experimental 
trials  involving  the  same  mode  of  automation  as  a  basis  for  triggering  DFAs  (manual  and 
automated  control  periods).  All  of  the  familiarization,  training,  and  practice  portions  of  the 
experiment  were  conducted  on  the  first  day  of  subject  participation. 

During  the  second  and  third  days  of  the  experiment,  subjects  reviewed  the  simulation 
procedures  and  completed  10  experiment  trials  involving  the  use  of  the  various  modes  of 
Multitask©  automation.  Each  of  the  trials  was  preceded  by  a  brief  review  of  the  mode  of 
automation  to  be  tested.  Each  subject  completed  two  trials  vmder  each  mode  of  automation. 
The  order  in  which  the  modes  were  presented  to  subjects  was  randomized  (see  Tables  3.7). 
Each  trial  lasted  approximately  50-min.,  including  30-min.  of  simulation  time  and 
approximately  20-min.  to  answer  all  three  SA  questionnaires. 

The  training,  practice,  and  experiment  sessions  were  conducted,  as  previously 
mentioned,  using  the  Multitask©  Skill  Level  of  seven  (i.e.,  7  targets  were  presented  on  the 
display  at  any  given  time).  The  setting  of  seven  targets  was  used  based  on  the  working 
memory  capacity  of  7,  plus  or  minus  2,  chunks,  as  defined  by  Miller  (1956).  All  training  and 
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practice  trials  presented  the  ATC  simulation  running  10-times  faster  than  real  time,  with  the 
exception  of  the  dual-task  and  SAGAT  practice  session,  and  the  experiment  trials,  which 
involved  the  Multitask©  running  2-times  faster  than  real  time.  These  simulation  speeds  were 
determined  appropriate  based  on  previous  research  conducted  by  Clamann  et  al.  (in  press) 
which  successfully  incorporated  similar  accelerated  speeds  for  training  and  testing. 
Furthermore,  pilot  testing  suggested  that  7  targets  traveling  at  2-times  faster  than  real  time 
was  an  appropriate  level  of  workload  for  experiment  trials. 

The  gauge-monitoring  task  was  used  as  an  index  of  workload  and  served  as  a  basis 
for  triggering  the  DFAs  during  AA  trials.  If  gauge  task  performance  dropped  below  the 
average  practice  performance  minus  1  SD  (as  determined  based  on  the  two  dual-task  plus 
SAGAT  practice  sessions  for  the  particular  mode  of  automation  being  tested)  for  one  minute 
of  a  trial  (implying  an  increase  in  subject  workload)  then  the  computer  initiated  a  switch 
from  manual  control  of  the  Multitask©  to  the  mode  of  automation  for  the  trial.  Therefore, 
the  form  of  AA  authority  used  in  this  study  was  computer  mandates  of  DFAs.  Once  the 
automation  was  initiated,  if  subject  performance  in  the  gauge  task  increased  above  the  mean 
plus  1  SD,  or  a  hit-to-signal  ratio  of  1.0  was  achieved  (perfect  performance)  for  one  minute 
of  a  trial,  then  manual  control  was  automatically  restored.  These  criteria  were  developed  by 
Kaber  and  Riley  (1999)  for  a  closed-loop  AA  system  and  validated  by  Clamaim  et  al.  (2002). 

The  dual-task  plus  SAGAT  practice  session  and  each  experiment  trial  incorporated  3 
SAGAT  freezes.  The  freezes  were  designed  to  occur  within  one  of  three  S-min  windows  of 
time  (7  to  14-min,  15  to  22-min,  and  23  to  30-min)  to  ensure  a  sampling  of  SA  throughout 
the  testing.  A  random  number  generator  was  used  to  generate  freeze  times  within  each  of  the 
windows.  Table  3.7  presents  the  minutes  into  a  tried  at  which  each  of  the  freezes  occurred. 
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The  first  six  minutes  were  not  considered  for  freezes  in  order  to  allow  subjects  time  to 
acquire  SA  (Endsley,  1995c),  and  to  allow  the  aircraft  time  to  move  from  the  radarscope 
periphery  towards  the  airports.  Furthermore,  no  two  freezes  were  scheduled  within  2-min.  of 
each  other  in  order  to  allow  subjects  time  to  reacquire  SA  after  a  stop  (Endsley,  1995c). 

Table  3.7  SAGAT  freeze  times 


Subject 

Mode 

Order 

Automation  Mode 

Timing  of  simulation  freezes 
(minutes  into  a  trial) 

Freeze  1 
(7-14  min) 

Freeze  2 
(15-22  min) 

Freeze  2 
(23-30  min) 

1 

NA 

Dual-task  and  SAGAT  Practice:  Info  Acq 

14 

20 

23 

10 

Manual- 1 

11 

16 

26 

5 

Manual-2 

7 

20 

23 

4 

Information  Acquisition- 1 

12 

20 

23 

6 

Information  Acquisition-2 

13 

15 

28 

3 

Information  Analysis- 1 

11 

21 

29 

S 

Information  Analysis-2 

7 

18 

25 

1 

Decision  Making- 1 

11 

19 

26 

9 

Decision  Making-2 

14 

19 

24 

7 

Action  Implementation- 1 

10 

15 

24 

2 

Action  Implementation-2 

11 

17 

23 

2 

NA 

Dual-task  and  SAGAT  Practice:  Info  Anal 

13 

19 

26 

2 

Manual- 1 

8 

22 

27 

7 

Manual-2 

8 

17 

8 

Information  Acquisition- 1 

13 

22 

26 

6 

Information  Acquisition-2 

9 

17 

26 

5 

Information  Analysis- 1 

7 

21 

25 

10 

Information  Analysis-2 

11 

21 

EEHHHIi 

9 

Decision  Making-1 

14 

18 

1 

Decision  Makmg-2 

9 

18 

28 

3 

Action  Implementation- 1 

9 

17 

28 

4 

Action  Implementation-2 

7 

21 

24 

3 

i 

i 

NA 

Dual-task  and  SAGAT  Practice:  Act  Imp 

14 

19 

28 

7 

Manual- 1 

8 

22 

24 

8 

Manual-2 

9 

26 

3 

Information  Acquisition- 1 

9 

16 

30 

4 

Information  Acquisition-2 

10 

22 

29 

6 

Information  Analysis-! 

10 

16 

25 

9 

Information  Analysis-2 

7 

22 

29 

10 

Decision  Making- 1 

13 

21 

24 

1 

Decision  Making-2 

11 

17 

30 

2 

Action  Implementation- 1 

16 

29 

5 

Action  Implementation-2 

10 

22 

30 
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Table  3.7  SAGAT  freeze  times  (continued) 


4 

NA 

Dual-task  and  SAGAT  Practice:  Act  Imp 

10 

17 

30 

2 

Manual- 1 

8 

15 

27 

1 

Manual-2 

10 

18 

3 

Information  Acquisition- 1 

11 

21 

29 

6 

Information  Acquisition-2 

9 

18 

26 

8 

Information  Analysis- 1 

13 

21 

28 

9 

Information  Analysis-2 

14 

16 

28 

4 

Decision  Making- 1 

7 

21 

27 

5 

Decision  Making-2 

14 

22 

24 

7 

Action  Implementation- 1 

9 

15 

24 

10 

Action  Implementation-2 

13 

17 

26 

5 

NA 

Dual-task  and  SAGAT  Practice:  Dec  Mak 

14 

17 

25 

3 

Manual- 1 

9 

23 

2 

Manual-2 

13 

18 

25 

8 

Information  Acquisition- 1 

14 

16 

30 

7 

Information  Acquisition-2 

13 

16 

23 

9 

Information  Analysis- 1 

9 

17 

25 

6 

Information  Analysis-2 

7 

21 

25 

Decision  Making- 1 

14 

19 

27 

4 

Decision  Making-2 

14 

21 

25 

1 

Action  Implementation- 1 

10 

22 

26 

5 

Action  Implementation-2 

14 

18 

29 

6 

7 

Dual-task  and  SAGAT  Practice:  Info  Acq 

8 

22 

29 

3 

Manual- 1 

13 

15 

23 

6 

Manual-2 

7 

19 

24 

9 

Information  Acquisition- 1 

14 

17 

25 

4 

Information  Acquisition-2 

14 

18 

25 

10 

Information  Analysis- 1 

9 

21 

2 

Information  Analysis-2 

14 

28 

8 

Decision  Making- 1 

9 

22 

25 

5 

Decision  Making-2 

13 

16 

23 

1 

Action  Implementation- 1 

8 

17 

28 

7 

Action  Implementation-2 

19 

29 

7 

9 

Dual-task  and  SAGAT  Practice:  Dec  Mak 

12 

21 

24 

4 

Manual- 1 

9 

18 

26 

7 

Manual-2 

10 

16 

23 

5 

Information  Acquisition- 1 

10 

16 

24 

10 

Information  Acquisition-2 

11 

19 

30 

1 

Information  Analysis- 1 

7 

16 

28 

6 

Information  Analysis-2 

12 

18 

26 

8 

Decision  Making- 1 

8 

15 

23 

3 

Decision  Making-2 

10 

15 

25 

2 

Action  Implementation- 1 

12 

18 

26 

9 

Action  Implementation-2 

13 

22 

24 
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Table  3.7  SAGAT  freeze  times  (continued) 


8 

5 

Dual-task  and  SAGAT  Practice:  Info  Anal 

9 

25 

10 

Manual-! 

12 

19 

24 

6 

Manuai-2 

9 

18 

26 

9 

Information  Acquisition-1 

7 

15 

27 

8 

Information  Acquisition-2 

14 

16 

29 

7 

Information  Analysis- 1 

11 

16 

24 

2 

Information  Analysis-2 

14 

16 

29 

3 

Decision  Making- 1 

13 

16 

26 

1 

Decision  Making-2 

8 

16 

24 

4 

Action  Implementation- 1 

13 

15 

27 

5 

Action  Implementation-2 

20 

26 

During  each  freeze,  subjects  were  posed  with  SA  queries  randomly  selected  from  a 
pool  of  1 8  total  SA  questions  targeting  the  three  levels  of  SA  defined  by  Endsley  (1995a;  see 
Appendix  B).  Each  freeze  included  6  questions.  At  the  beginning  of  a  freeze,  subjects  were 
asked  to  identify  the  locations  of  the  7  aircraft  on  the  radarscope  by  labeling  a  drawing  of  the 
radarscope  with  the  numbers  1-7.  Subsequently,  they  were  asked  to  respond  to  each  of  the  6 
SA  queries  for  each  aircraft  by  completing  tables  like  those  presented  in  Appendix  B.  The 
questions  were  administered  electronically  using  a  computer  database  application.  While 
responding  to  SA  questions,  subjects  were  allowed  to  review  question  descriptions  (see 
Appendix  G),  but  the  Multitask©  display  was  not  visible  to  subjects  during  queries. 

Since  Multitask©  is  truly  an  AA  simulation  in  which  a  real-time  workload  measure  is 
used  to  trigger  DFAs  between  manual  and  automation  control,  it  is  not  possible  to  predict  at 
which  moments  during  a  simulation  trial  a  subject  will  be  using  manual  or  automated  control. 
Consequently,  it  was  not  possible  to  pre-determine  a  distribution  of  SAGAT  stops  during 
trials  that  could  produce  an  even  number  of  SA  queries  under  manual  and  automated  control 
conditions.  However,  since  the  SAGAT  freeze  times  were  randomly  determined,  there  was 
an  equal  likelihood  that  the  S  A  queries  were  posed  during  Multitask©  manual  control 
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periods  or  automated  control  periods.  Since  a  large  number  of  SAGAT  freezes  were 
conducted  during  the  experiment,  the  average  percent  correct  responses  to  SA  queries  under 
manual  control  or  automated  control  was  expected  to  be  representative  of  actual  operator  SA. 

The  entire  experiment  lasted  three  weeks.  Each  subject  was  tested  for  approximately 
13  hours  over  three  separate  days.  The  day-of-the-week  and  time  of  each  subject’s 
participation  remained  consistent  during  the  experiment  with  the  exception  of  two  scheduling 
conflicts.  At  the  start  and  conclusion  of  each  day  of  testing,  a  payment  form  was  completed 
and  verified  by  the  subject  (see  Appendix  E).  At  the  conclusion  of  the  third  day  of  testing, 
each  subject  signed  the  payment  form,  and  they  were  debriefed.  In  addition,  subjects 
completed  a  Closing  Questionnaire  (see  Appendix  H).  A  more  detailed  description  of  the 
experimental  procedmes  is  presented  in  Appendix  I. 

3.7  Hypotheses 

Adaptive  automation  was  expected  to  have  a  significant  effect  on  ATC  simulation 
(Multitask©)  performance.  It  was  expected  that  during  the  manual  trials.  Multitask© 
performance  would  be  worse  than  during  any  mode  of  automation.  This  would  be  consistent 
with  findings  presented  by  Clamann  et  al.  (2002)  who  observed  that  any  AA  of  ATC 
information  processing  functions  produced  better  performance  than  no  automation 
whatsoever.  In  addition,  as  Leroux  (1993),  Laois  and  Giannacourou  (1995),  and  Kaber  et  al. 
(2002)  found,  subjects  have  performed  better  using  automation  that  provides  assistance  with 
low-level  sensory/response  functions,  such  as  information  acquisition  and  action 
implementation.  Better  task  performance  was  also  expected  in  this  study  during  trials 
applying  AA  to  these  low-level  HIP  functions. 
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Adaptive  automation  was  also  expected  to  affect  performance  on  the  secondary, 
gauge-monitoring  task,  or  operator  workload.  Modes  of  automation  that  require  high  levels 
of  attentional  resources  may  increase  simulation  workload,  evidenced  by  decreases  in  gauge¬ 
monitoring  task  performance.  For  example,  the  information  analysis  mode  of  automation, 
though  implemented  to  aid  the  user  and  improve  simulation  performance,  presents  a  large 
amount  of  data  and  was  expected  to  increase  controller  workload  (Kaber  et  al.,  2001). 
Likewise,  modes  of  automation  that  do  not  present  a  great  deal  of  information  (e.g.,  action 
implementation)  or  present  the  information  directly  on  the  radar  display  (e.g.,  information 
acquisition)  were  expected  to  decrease  workload.  Furthermore,  an  overarching  goal  of 
implementing  AA  is  to  reduce  operator  workload  compared  to  operating  under  full 
automation  or  manual  control  (Hilbum  et  al.,  1997).  Therefore,  the  experimental  trials 
involving  manual  control  were  expected  to  result  in  higher  levels  of  operator  workload  than 
the  AA  trials. 

Finally,  AA  was  expected  to  have  an  effect  on  operator  SA.  As  Endsley  (1995a)  and 
Wickens  (2000)  observed,  high  levels  of  automation  may  result  in  OOTL  performance,  and 
can  produce  lower  levels  of  operator  SA.  Therefore,  subjects  were  expected  to  do  better  at 
responding  to  SAGAT  queries  posed  during  the  manual  trials  of  this  experiment  and  manual 
control  minutes  of  an  AA  trial  than  during  periods  of  automation.  Subjects  were  also 
expected  to  perform  better  on  SAGAT  queries  during  modes  of  automation  that  maintain 
user  involvement  in  the  system,  like  information  acquisition  (Kaber  et  al.,  1999).  With 
respect  to  action  implementation  automation,  this  mode  automatically  hands-off  aircraft  to 
tower  control,  eliminating  the  risk  of  a  collision  within  20  nm  of  the  radarscope  center.  This 
mode  was  expected  to  improve  Multitask©  performance,  but  it  also  removes  the  controller 
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from  the  control  loop  and  was  expected  to  adversely  affect  SA.  In  addition,  Kaber  and 
Endsley  (1997)  found  that  low-level  automation  involving  batch  processing  of  operator  task 
strategies  lead  to  decreases  in  SA  in  comparison  to  other  forms  of  automation  maintaining 
operators  in  active  control  of  the  system. 

Beyond  this,  the  percent  correct  responses  to  Level  1  SA  questions  were  expected  to 
deteriorate  during  trials  that  involved  the  information  acquisition  LOA,  because  the  subject 
was  further  removed  from  the  system  control  loop  in  terms  of  gathering  data  on  aircraft. 
Likewise,  Level  2  SA  query  performance  (pertaining  to  the  comprehension  stage  of  HIP)  was 
expected  to  be  worse  under  the  information  analysis  LOA,  because  under  that  mode  of 
automation  the  operator  was  removed  from  the  process  of  integrating  aircraft  information  to 
identify  conflicts.  In  addition,  decision  making  automation  trials  were  expected  to  decrease 
the  accuracy  of  responses  to  Level  3  SA  questions,  because  active  involvement  in  the 
decision  process  may  affect  a  controller’s  ability  to  predict  the  future  status  of  the  simulation. 
Furthermore,  decreases  in  the  accuracy  of  responses  to  Level  1  and  Level  2  SA  questions 
were  expected  to  decrease  Level  3  SA  accuracy,  because  the  projection  of  future  system 
status  depends  on  accurate  information  perception  and  comprehension  (Endsley  &  Rodgers, 
1994;  Endsley,  1995a). 
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4  Data  Analysis 


The  experiment  followed  a  repeated  measures  design  (or  randomized  complete  block 
design  with  blocking  on  the  subject)  in  which  each  subject  completed  multiple  trials  under 
each  level  of  the  IV.  As  stated,  the  order  of  presentation  of  modes  of  automation  to  subjects 
was  randomized.  This  design  allowed  each  experimental  unit  to  be  used  to  compare  the 
mode  of  automation  effects  on  the  various  response  measures  and  controlled  for  the 
variability  among  subjects  (Montgomery,  2001,  p.  127). 

Analyses  of  Variance  (ANOVAs)  were  conducted  on  all  response  measures  (primary 
task  performance,  gauge-monitoring  task  performance,  and  SA)  with  the  mode  of  AA  and 
subject  (to  identify  individual  differences)  included  as  predictor  variables  in  the  ANOVA 
model.  Type  III  sums  of  squares  were  used  to  determine  the  significance  of  the  IV  on 
responses  (SAS  Institute  Inc.,  1990,  p.  120-21).  The  mean  square  error  (MSE)  for  the 
specific  statistical  models  was  used  in  all  F-tests  on  the  IV.  For  a  repeated  measures  design 
with  a  single  treatment,  and  the  subject  term  included  in  the  statistical  model,  Montgomery 
(2001,  p.  624)  states  that  this  is  the  appropriate  form  of  the  F-test  and  that  such  a  model 
partitions  the  variance  attributable  to  the  subject.  (A  conservative  approach  to  assessing  the 
significance  of  a  treatment  in  human  factors  experiments  is  to  use  the  mean  square  for  the 
subject  variable  as  a  denominator  in  the  F-test.) 

Finally,  post-hoc  tests  were  conducted  using  Tukey’s  honestly  significant  differences 
test  to  determine  which  mode  of  AA  produced  superior  results  in  terms  of  primary  task 
performance,  workload  and  SA.  An  alpha  level  of  0.05  was  used  to  identify  any  significant 
effect  of  the  mode  of  automation. 
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Multitask©  Data  Analysis 

In  order  to  make  comparisons  of  the  AA  conditions  with  the  manual  control 
condition,  the  total  nmnber  of  aircraft  presented  in  each  trial  was  determined  based  on  the 
number  of  aircraft  contacted  (or  slewed)  by  a  subject,  and  was  used  as  a  denominator  in 
calculating  ratios  of  aircraft  cleared,  conflicts,  and  collisions  to  total  aircraft  (i.e.,  cleared 
aircraft/aircraft  slewed).  This  was  done  to  account  for  any  potential  differences  in  the  total 
number  of  aircraft  presented  across  conditions. 

The  analyses  of  the  response  observations  collected  during  the  manual  and  automated 
control  periods  as  part  of  the  AA  conditions  were,  however,  not  conducted  in  this  manner, 
because  the  total  number  of  aircraft  generated  during  manual  or  automated  control  periods 
could  not  be  determined  on  the  basis  of  subject  slewing  of  aircraft.  It  is  important  to  recall 
here  that  there  were  always  7  aircraft  on  the  simulation  display  at  any  given  time,  and  the 
nature  of  the  AA  algorithm  as  part  of  the  Multitask©  simulator  allowed  for  comparable 
amounts  of  manual  and  automated  control  time  across  modes  of  automation  (although  the 
exact  proportions  of  manual  and  control  periods  is  unpredictable).  For  this  reason,  the 
performance  measures  on  the  manual  and  automated  control  periods  (as  part  of  the  AA  trials) 
analyzed  in  this  study  consisted  of  the  total  number  of  cleared  aircraft,  conflicts,  and 
collisions. 

Two  data  points  were  removed  fi-om  the  aircraft  conflict  data  as  the  result  of  a  subject 
failing  to  follow  instructions  during  two  trials  (Subject  7,  Trials  3  and  4).  This  reduced  the 
number  of  observations  on  conflicts  for  the  experiment  firom  80  to  78  and  decreased  the 
number  of  conflict  observations  on  the  manual  and  automated  control  periods  during  the  AA 
trials  from  64  to  63  (one  of  the  trials  was  a  manual  control  condition  trial).  Two  data  points 
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were  also  removed  from  the  collision  data  set  as  the  result  of  an  aircraft  appearing  on  the 
radar  display  at  exactly  the  same  location  as  another  aircraft  (Subject  2,  Trial  1 ;  Subject  5, 
Trial  5).  These  two  occurrences  were  considered  to  be  problems  with  the  simulation. 

Workload  Data  Analysis 

Subject  workload  was  objectively  measured  in  terms  of  the  hit-to-signal  ratio  in  the 
gauge-monitoring  task.  Average  ratios  were  calculated  for  each  test  trial  (including  manual 
and  automated  control  periods)  and  separate  mean  hit-to-signal  ratios  were  calculated  on 
performance  during  manual  and  automated  control  periods  as  part  of  the  AA  conditions.  One 
observation  was  missing  from  the  gauge-monitoring  task  data  as  the  result  of  one  (very  short) 
automated  control  period  not  including  any  gauge  pointer  deviations  (Subject  1,  Trial  8). 

This  decreased  the  number  of  observations  for  automation  control  periods  to  63. 

Diagnostics  and  Statistical  Model 

Subsequent  to  organizing  the  primary  task  and  workload  performance  data  into  three 
data  subsets  (total  performance,  performance  during  automated  control,  and  performance 
during  manual  control),  data  analyses  were  conducted  using  SAS.  ANOVA  model  residual 
values  were  output  and  plotted  against  the  model  predicted  values,  mode  of  automation,  and 
trial  number  to  determine  if  the  data  met  the  assumptions  of  the  ANOVA  including  random 
process,  linearity,  and  constant  variance.  The  normality  assumption  was  also  assessed  using 
Shapiro-Wilk’s  test  and  normal  probability  plots.  These  diagnostics  indicated  if  any 
transformations  on  the  response  or  IV  might  be  required  to  ensure  the  appropriateness  of  the 
parametric  test,  as  prescribed  by  Neter,  Wasserman,  and  Kutner  (1990).  Any  transformations 
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employed  in  the  various  analyses  are  described  in  Section  5.  Based  on  the  residual  and  trial 
number  plots  no  learning  was  found  in  either  the  Multitask©  or  gauge-monitoring 
performance  data  sets. 

A  statistical  model  in  mode  of  automation  and  subject  was  used  to  assess  the  impact 
of  the  various  AA  conditions  on  response: 

Yij  =  p.  +  Ai  +  Sj  +  s  (Equation  5.1) 

for 

i=l,2, 3,4,5 
j  =  l,2,3,4,5,6,  7,8 

where 

Yij  =  Response  measure 
Ai  =  Mode  of  automation 
Sj  =  Subject 
8  =  Experimental  error 

Situation  Awareness  Data  Analysis 

On  the  basis  of  subject  responses  to  the  SA  queries,  the  average  percentage  of  correct 
responses  to  Level  1, 2,  and  3  questions,  and  a  total  SA  score,  were  determined  for  each  stop. 
The  type  of  simulation  control,  manual  or  automated  as  part  of  the  AA  conditions,  was  also 
recorded  at  each  stop,  and  an  analysis  was  conducted  to  identify  any  significant  effects  on 
SA.  As  previously  mentioned,  six  of  the  18  SAGAT  questions  were  randomly  assigned  to 
each  SAGAT  stop  resulting  in  240  observations  for  the  entire  experiment  (8  subjects  x  10 
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trials  X  3  stops).  However,  various  oceurrences  resulted  in  the  collection  of  less  than  the 
expected  240  observations: 

(a)  based  on  responses  to  the  SA  questions,  it  was  determined  that  Subject  1  did  not 
understand  Question  7  and  Subject  8  did  not  understand  Question  1 1  (see  Appendix 
G  for  question  descriptions); 

(b)  some  questions  were  not  answered  as  a  result  of  subjects  inadvertently  pressing  a 
key  on  die  computer  keyboard  or  subjects  deliberately  closing  the  question  window 
because  they  did  not  know  any  of  the  answers; 

(c)  some  SAGAT  stops  occurred  in  the  last  minutes  of  the  simulation  and  posed 
questions  for  which  the  true  answers  were  indeterminable  (see  Questions  14  and  17  in 
Appendix  G);  and 

(d)  due  to  the  random  assignment  of  the  18  questions  to  trial  stops,  some  stops  did  not 
incorporate  any  questions  representing  one  of  the  three  SA  levels. 

Subsequent  to  an  analysis  of  the  SAGAT  data  to  determine  any  type  of  control 
(manual  or  automation)  effects,  the  percent  correct  responses  to  Level  1, 2,  and  3  SA  queries, 
and  total  S  A  for  each  trial  were  analyzed  for  potential  effects  of  the  mode  of  automation. 

The  percent  correct  responses  to  each  individual  SA  question  were  also  analyzed  for 
sensitivity  to  the  automation  manipulation.  As  discussed  previously,  this  approach  to 
SAGAT  data  analysis  has  been  used  by  many  researchers  (Endsley,  1995b;  Endsley  &  Kiris, 
1995;  Bolstad  &  Endsley,  2000;  Hauss  &  E5rferth,  2003;  Jones  and  Kaber,  in  press)  and  is  an 
appropriate  measure  of  S  A. 
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Each  SA  question  at  each  stop  solicited  7  responses  from  subjects  (a  response  was 
required  for  each  of  the  7  aircraft  on  the  display;  see  Appendix  B);  however,  less  than  the  7 
possible  responses  may  have  occurred  during  certain  trials  for  the  following  reasons: 

(a)  some  subjects  were  not  able  to  correctly  identify  all  7  aircraft  on  the  SAGAT  key 
(see  the  radar  scope  diagram  in  Appendix  B)  during  simulation  freezes; 

(b)  some  subjects  did  not  respond  to  a  particular  question  for  all  aircraft; 

(c)  the  true  answers  for  certain  questions  were  not  obtainable  for  some  aircraft  on  the 
display  (e.g.,  minutes  until  destination  is  unknown  when  a  trial  ends  with  an  aircraft 
in  a  holding  pattern);  and 

(d)  some  aircraft  had  not  yet  been  contacted  when  freezes  occurred,  therefore  some 
information  was  not  yet  available  to  the  subject  (e.g.,  destination  airport). 

These  instances  necessitated  that  the  SA  responses  were  scored  as  a  ratio  of  correct  answers 
to  possible  responses  (i.e.,  the  number  of  correctly  identified  aircraft  was  used  as  the 
denominator  for  SA  question  scores.) 

Since  the  responses  to  SAGAT  questions  represent  a  binomial  variable  (correct  or 
incorrect)  the  discrete  nature  of  the  responses  violates  the  ANOVA  assumptions.  With  this 
in  mind,  Endsley  (1995c)  found  the  arcsine  function  (e.g.,  Y’=arcsin(Y))  to  be  an  effective 
transformation  to  account  for  this  problem  in  her  use  of  SAGAT.  The  arcsine  function  was 
applied  to  the  percentages  for  each  question,  and  the  effectiveness  of  the  transformation  was 
verified  using  the  same  diagnostic  plots  and  tests  considered  in  evaluating  the  performance 
data. 
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5  Results 


5.1  Primary  Task  Performance  (Multitask©) 

Since  significant  individual  differences  were  revealed  through  ANOVA  results  on  a 
number  of  the  response  measures  (primary  task  performance,  workload  and  SA),  power 
analyses  were  conducted  for  all  statistical  tests  in  order  to  provide  some  evidence  of  whether 
the  sample  size  might  have  been  influential  in  the  findings.  The  p  values  for  each  statistical 
test  are  reported  along  with  the  test  statistics  and  significance  levels.  Furthermore,  power 
values  are  reported  for  specific  insignificant  results  that  were  counter  to  expectation.  In 
general,  it  is  possible  that  the  sample  size  for  the  experiment  may  have  limited  the  sensitivity 
of  some  of  the  analyses  for  revealing  true  difference  among  the  automation  conditions. 

Total  Performance 

The  mode  of  Multitask©  AA  was  expected  to  effect  subject  performance,  however  an 
ANOVA  on  total  Multitask©  performance  (during  manual  and  automated  periods  combined) 
revealed  no  significant  effects  of  the  mode  of  automation  on  the  ratio  of  cleared  aircraft, 
aircraft  conflicts,  and  aircraft  collisions  to  aircraft  generated  (p>0.35,  P>0.60).  This  was 
surprising  as  7  out  of  the  8  subjects  (88%)  commented  on  closing  questionnaires  that 
operating  the  simulation  was  easier  when  automation  was  provided. 

Automated  Control  Periods 

With  respect  to  the  automated  control  periods  as  part  of  the  AA  conditions,  an 

ANOVA  revealed  a  significant  effect  of  mode  of  automation  (F(3,53)=4.03,  p=0.01 18, 
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P=0.54)  on  the  number  of  cleared  aircraft.  There  also  appeared  to  be  significant  individual 
differences  (F(7,53)=  10.31,  p<0.0001).  Analysis  of  Variance  results  also  revealed  the 
number  of  aircraft  collisions  during  automated  control  periods  to  be  significantly  affected  by 
the  mode  of  automation  (F(3,51)=3.02  p=0.0382,  p=0.33).  The  initial  analysis  of  conflicting 
aircraft  during  automation  control  periods  revealed  a  significant  Shapiro- Wilk’s  test 
(p=0.0127)  and  a  departure  of  the  data  from  a  linear  trend  in  the  normal  probability  plot, 
suggesting  a  violation  of  the  ANOVA  normality  assumption.  Consequently,  a  Logic 
transformation  was  applied  to  the  response  measures  (Neter,  Wasserman,  &  Kutner,  1990, 
p.l46)  attempting  to  account  for  the  apparent  normality  violation.  An  ANOVA  on  the  Logic 
transform  of  the  number  of  aircraft  conflicts  revealed  no  significant  effect  of  the  mode  of 
automation  (p>0.60,  p>0.85).  Figure  5.1  summarizes  the  mean  number  of  aircraft  cleared, 
conflicting,  and  colliding  under  each  mode  of  AA  during  the  automation  control  periods  of 
the  Multitask©  simulation. 
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Figure  5.1  Primary  task  performance  during  automation  control  periods 

It  is  apparent  from  the  plot  that,  as  hypothesized,  the  action  implementation  mode  of 
automation,  a  low-level  sensory/response  function,  yielded  higher  average  performance 
during  automated  control  periods  in  terms  of  cleared  aircraft  due  to  the  automated  hand-off 
of  aircraft  control  from  TRACON  control  to  local  tower  control.  A  Tukey’s  test  confirmed 
that  action  implementation  was  significantly  better  (p<0.05)  in  terms  of  cleared  aircraft,  as 
compared  to  the  information  analysis  mode  of  automation,  a  high-level  air  traffic  controller 
cognitive  function.  Tukey’s  test  also  revealed  information  analysis  to  be  significantly 
inferior  (p<0.05)  for  preventing  collisions  compared  to  decision  making  automation,  further 
supporting  the  hypothesis  that  AA  is  most  effective  when  applied  to  low-level  cognitive 
functions. 
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Manual  Control  Periods 


Residual  diagnostics  for  the  cleared  aircraft  data  collected  during  manual  control 
periods  of  the  AA  trials  revealed  non-constant  variance  across  the  modes  of  automation.  A 
square  root  transformation  was  performed  on  the  response  data  (Neter,  Wasserman,  & 

Kutner,  1990,  p.l46)  in  order  to  accoimt  for  the  potential  ANOVA  assumption  violation. 
Analysis  of  Variance  results  revealed  a  significant  effect  of  mode  of  automation 
(F(3,53)=3.73,  p=0.0166,  P=0.64)  on  the  square  root  of  cleared  aircraft.  There  were  also 
highly  significant  individual  differences  (F(7,53)=8.39,  p<.0001)  relating  to  cleared  aircraft 
during  manual  control  periods.  Mode  of  automation  did  not  prove  to  have  a  significant  effect 
on  the  number  of  aircraft  conflicts  (p>0.75,  p>0.85).  An  analysis  of  model  residuals  for  the 
number  of  collisions  during  manual  control  periods  revealed  a  significant  Shapiro- Wilk’s  test 
(p<0.0001)  and  a  departure  of  the  data  from  the  ANOVA  normality  assumption  based  on  a 
non-linear  normal  probability  plot.  A  square  root  transformation  was  applied  to  the  response 
measure  (Neter,  Wasserman,  &  Kutner,  1990,  p.l46)  which  improved  the  conformance  of  the 
data  with  the  normality  assumption.  However,  the  transformed  data  revealed  no  significant 
effect  of  the  mode  of  automation  on  the  square  root  of  aircraft  collisions  during  manual 
control  periods  (p>0.65,  P>0.85).  Figure  5.2  shows  the  mean  values  for  the  number  of 
aircraft  cleared,  conflicting,  and  colliding  during  manual  control  periods  of  the  AA 
simulation  trials. 
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Figure  5.2  Primary  task  performance  during  manual  control  periods 

Contrary  to  hypotheses,  the  plot  shows  higher  average  performance  during  trials  involving 
the  higher-level  information  analysis  mode  of  AA  in  terms  of  cleared  aircraft.  A  Tukey’s  test 
confirmed  that  the  information  analysis  mode  of  automation  was  significantly  better  for 
clearing  aircraft  (p<0.05)  than  the  action  implementation  mode  of  automation.  This  carry¬ 
over  effect  is  discussed  in  Section  6. 


5.2  Secondary  Gauge-monitoring  Task  Performance 

Performance  observations  collected  on  the  secondary  task  (workload)  were  organized 
into  the  same  data  subsets  as  the  primary  task  performance  data;  performance  recorded 
during  both  manual  and  automated  control  periods,  performance  during  automation  control 
periods  as  part  of  AA  trials,  and  manual  control  performance  as  part  of  the  AA  conditions. 
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An  ANOVA  on  the  workload  data  did  not  reveal  any  significant  effects  due  to  the 
mode  of  automation  when  comparing  the  manual  control  condition  with  the  AA  conditions 
(p>0.95,  p>0.90)  or  when  analyzing  the  automated  and  manual  control  periods  as  part  of  AA 
(p>0.75,  P>0.85  and  p>0.10,  p>0.70  respectively).  These  results  are  surprising  as  the  modes 
of  automation  were  expected  to  decrease  workload  (increased  average  secondary-task 
performance)  compared  to  the  manual  control  condition.  Likewise,  the  information 
acquisition  and  action  implementation  modes  of  automation  were  expected  to  decrease 
workload  compared  to  the  other  modes  of  automation.  Closing  questionnaire  responses 
identified  information  acquisition  as  the  preferred  mode  of  automation  for  6  out  of  the  8 
subjects  (75%).  There  were  significant  individual  differences  in  secondary-task  performance 
revealed  through  the  comparison  of  the  completely  manual  condition  with  AA  conditions 
(F(7,68)= 16.97,  p<0.0001),  and  in  analyzing  the  automated  control  periods  (F(7,52)=8.19, 
p<0.0001)  and  manual  control  minutes  (F(7,68)=l  1.89,  p<0.0001)  separately. 

5.3  SAGAT  Performance 

Automation  Versus  Manual  Control  Results 

The  percent  correct  SAGAT  responses  at  each  freeze  were  analyzed  to  determine  the 
effect  of  simulation  control  type  (automated  or  manual  control  periods)  on  SA.  As  expected, 
an  ANOVA  revealed  a  significant  effect  of  the  general  control  type  on  the  arcsine  of  percent 
correct  responses  to  Level  3  SA  queries  (F(l,216)=9.33,  p=0.0025,  p=0.75);  however,  there 
were  no  significant  effects  of  the  general  control  type  on  the  percent  correct  responses  to 
Level  1  and  2  SA  queries  or  for  the  total  SAGAT  scores  (p>0.20,  p>0.90),  although  there 
were  significant  individual  differences  (F(7,216)=5.11,  p<.0001).  Figure  5.3  summarizes  the 
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mean  Level  1,  Level  2,  Level  3  and  total  SA  scores  for  both  automated  and  manual  control 


periods. 


Figure  5.3  Mean  SAGAT  scores  during  manual  and  automation  control  periods 

As  hypothesized,  the  plot  reveals  that  subjects  were,  on  average,  better  at  answering  Level  3 
SAGAT  queries  during  manual  control  periods  compared  to  automated  control  periods.  In 
addition,  closing  questionnaire  results  revealed  that  75%  of  subjects  found  answering  the  SA 
questions  easier  while  the  simulation  was  operating  imder  manual  control. 

Results  on  Aggregate  SA  Measures 

Surprisingly,  none  of  the  aggregate  SA  response  measures  revealed  significance  of 
the  mode  of  automation  (p>0.65,  p>0.80).  This  was  unexpected  because  the  classification  of 
various  queries  according  to  Endsley’s  (1995a)  levels  of  SA  was  considered  to  be  accurate 
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based  on  the  type  of  dynamic  knowledge  the  question  called  for.  Further  discussion  of  these 
results  is  presented  in  subsequent  sections. 

Individual  Results  on  SA  Queries 

The  SAGAT  data  was  then  analyzed  on  a  question  by  question  basis,  resulting  in  18 
separate  analyses.  Most  surprisingly,  ANOVA  results  revealed  no  significant  effect  of  mode 
of  automation  on  the  arcsine  transform  of  percent  correct  responses  for  all  questions, 
although  significant  individual  differences  were  observed  for  SA  responses  to  Questions  1, 3, 
4,  5, 7, 10, 13, 14, 15, 16, 17,  and  18  (see  Appendix  G  for  question  descriptions).  It  was 
expected  that  SA  would  decrease  during  action  implementation  as  the  controller  may  have 
been  OOTL  compared  to  other  modes  of  automation.  In  addition,  the  percent  correct 
responses  to  Level  1  SA  questions  were  expected  to  deteriorate  during  trials  that  involved  the 
information  acquisition  LOA,  Level  2  S A  was  also  expected  to  be  worse  imder  the 
information  analysis  LOA,  and  decision  making  automation  trials  were  expected  to  decrease 
the  accuracy  of  responses  to  Level  3  SA  questions.  No  significant  findings  were  revealed  for 
these  analyses.  This  was  unexpected  because  previous  research  on  SA  in  ATC  has 
demonstrated  SAGAT  to  be  a  valid  and  reliable  indicator  of  controller  perception, 
comprehension,  and  projection  for  varying  workload  and  display  conditions  (Endsley  & 
Kiris,  1995;  Endsley,  Sollenberger,  Nakata,  &  Stein,  2000).  However,  some  more  recent 
work  by  Hauss  and  Eyferth  (2003)  suggests  that  SAGAT  may  not  be  a  sensitive  measure  for 
S  A  in  the  ATC  environment  due  to  different  aircraft  having  different  relevance  to  controllers 
at  different  times  in  a  simulation.  This  and  other  potential  shortcomings  of  the  SAGAT 
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measurement  approach  are  addressed  in  the  Discussion  section  in  explaining  the  lack  of 
sensitivity  of  the  measure  in  this  study. 
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6  Discussion 


6.1  Primary  Task  Performance  (Multitask©) 

Total  Performance 

It  was  expected  that  there  would  be  significant  performance  differences  among  the 
modes  of  AA  based  on  the  iinique  functionality  of  the  various  automation  aids  applied  to  the 
ATC  information  processing  functions.  In  addition,  a  significant  difference  was  expected 
between  the  four  modes  of  AA  and  the  manual  condition,  as  observed  by  Kaber  et  al.  (2001) 
and  Clamann  et  al.  (2002).  However,  there  was  no  significant  effect  of  automation  versus 
manual  control,  in  general,  on  Multitask©  performance.  One  explanation  for  this  is  the 
disproportionate  number  of  manual  and  automated  control  periods  during  the  AA  trials.  The 
AA  trials  involved  shifts  between  automated  and  manual  control  based  on  operator  workload 
states,  and  the  information  acquisition,  information  analysis,  and  decision  making  modes  of 
AA  all  involved  more  manual  control  minutes  than  automated  control  minutes  (59%,  63%, 
and  60%  manual  minutes,  respectively).  It  is  possible  that  the  large  numbers  of  manual 
control  minutes  as  part  of  the  AA  conditions  may  have  caused  any  differences  between  the 
modes  of  automation  and  the  manual  condition  to  be  indiscernible  by  making  them  all 
perform  more  like  a  manual  trial. 

Automated  Control  Periods 

During  automated  control  periods,  subjects  produced  superior  primary  task 

performance  in  terms  of  cleared  aircraft  during  the  action  implementation  trials.  In  addition, 

the  information  analysis  mode  of  automation,  a  high-level  fimction  including  a  complex 
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information  display,  produced  the  highest  number  of  aircraft  collisions.  These  findings  are 
consistent  with  the  results  of  Leroux  (1993),  Laois  and  Giannacourou  (1995),  Kaber  et  al. 
(2002),  and  Clamann  and  Kaber  (in  press)  who  found  automation  applied  to  low-level 
sensory/response  fimctions  to  be  most  beneficial  when  compared  to  complex  cognitive 
functions. 

Manual  Control  Periods 

Although  average  subject  performance  was  significantly  worse  during  automated 
control  periods  as  part  of  the  information  analysis  condition,  there  appeared  to  be  a  positive 
carry-over  effect  of  automation  on  subject  ability  to  clear  aircraft  during  manual  control 
periods  as  part  of  the  same  condition.  This  finding  suggests  that  information  analysis 
automation  may  have  better  prepared  subjects  for  manual  control  periods  compared  to  the 
other  forms  of  AA.  However,  tiiis  was  not  the  case  for  subject  ability  to  deal  with  negative 
events  including  conflicts  and  collisions.  On  average,  there  was  a  greater  number  of  conflicts 
and  collisions  during  the  manual  periods  of  the  information  analysis  condition,  but  any 
differences  among  the  AA  conditions  were  not  statistically  significant.  These  findings 
relating  to  the  information  analysis  mode  of  automation  during  manual  control  periods  could 
be  attributed  to  the  fact  that  the  average  duration  of  manual  control  periods  within  this 
condition  was  larger  than  any  other  AA  condition  (remember,  the  performance  measures  for 
manual  and  automated  control  periods  were  recorded  as  the  number  of  measure  occurrences.) 
The  lack  of  additional  significant  findings  on  the  manual  control  periods  may  be  attributed  to 
the  nature  of  manual  control  remaining  consistent  across  the  various  modes  of  AA.  Clamann 
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&  Kaber  (in  press)  also  observed  no  significant  findings  relating  to  manual  control  periods  of 
AA. 

Defining  Multitask©  Performance  Measures 

The  aircraft  conflict  performance  measure  did  not  reveal  any  significant  effect  of  the 
mode  of  automation.  Since  the  subjects  in  this  study  were  naive  students,  their  motivation 
for  maintaining  aircraft  separation  may  have  been  far  less  than  would  be  expected  with  actual 
controllers.  Furthermore,  the  assessmait  of  conflicts  depended  on  an  in-depth  understanding 
of  the  simulation,  and  subject  ability  to  mentally  picture  aircraft  separation  boundaries  during 
the  simulation.  This  was  a  complex  task  as  part  of  the  simulation.  In  regards  to  real  ATC 
performance  and  safety,  the  aircraft  conflict  measure  is  a  derivative  of  airspace  regulations, 
and  an  aircraft  conflict  simply  describes  two  aircraft  that  are  flying  in  close  proximity  to  each 
other.  Therefore,  two  aircraft  that  are  conflicting  within  a  real  ATC  environment  have  not 
physically  contacted  each  other  nor  have  the  people  onboard  the  aircraft  been  harmed.  By 
comparison,  aircraft  collisions  almost  certainly  result  in  loss  of  life  or  injury  to  both 
passengers,  crew  members,  and  people  on  the  ground  in  addition  to  immense  financial  loss. 

It  was  clear  to  all  subjects  in  this  study  when  a  collision  was  imminent  (conflict)  and  when 
one  occurred.  Similarly,  the  aircraft  cleared  performance  measure  is  a  direct  indicator  of 
controller  efficiency.  Aircraft  cleared  and  aircraft  collision  performance  measures  are  the 
most  crucial  to  ATC,  and  the  AA  conditions  significantly  affected  both  of  these  measures  in 
this  study. 
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6.2  Secondary  Gauge-monitoring  Task  Performance 

Based  on  the  results  presented  by  Kaber  et  al.  (2002),  the  AA  conditions  were 
expected  to  affect  operator  average  workload.  In  their  research,  they  observed  higher  levels 
of  workload  when  AA  was  applied  to  the  information  analysis  function  of  the  primary  task 
compared  to  the  information  acquisition  and  action  implementation  modes  of  automation. 
Furthermore,  as  Hilbum  et  al.  (1997)  suggest,  experiment  trials  involving  the  Tnanual  control 
condition  were  expected  to  increase  workload  compared  to  the  AA  conditions,  as 
implementing  AA  reduces  operator  workload  compared  to  operating  under  full  automation  or 
manual  control.  However,  the  mode  of  Multitask©  automation  and  the  comparison  of  the 
manual  condition  to  the  AA  conditions  in  this  study  did  not  reveal  any  significant  differences 
in  terms  of  gauge  monitoring  performance.  Anecdotal  observations  during  the  experiment 
and  subject  responses  to  the  closing  questionnaire  suggested  high  workload  associated  with 
the  information  analysis  mode  of  automation  due  to  the  large  amoimt  of  data  displayed 
during  these  trials.  Related  to  this,  information  acquisition  was  identified  as  the  preferred 
mode  of  automation  during  closing  questionnaires.  However,  there  was  no  statistical 
evidence  to  support  these  suggestions.  This  lack  in  significant  findings  may  be  attributed  to 
varying  strategies  employed  by  subjects  to  monitor  the  secondary  task.  Significant 
individual  differences  support  this  assertion.  In  general,  these  statistical  findings  presented 
here  are  consistent  with  Clamann  and  Kaber’s  (in  press)  findings  that  the  type  of  automation 
does  not  sigmficantly  affect  secondary-task  performance  (workload)  during  mariual  and 
automated  control  periods  as  part  of  AA  applied  to  ATC-like  tasks. 
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6.3  Situation  Awareness 


Automated  Versus  Manual  Control  Effects 

As  hypothesized,  subjects  were  better  at  projecting  the  future  status  of  the  Multitask© 
air  traffic  (Level  3  SA)  during  manual  control  periods  of  the  simulation  as  opposed  to 
automated  control  periods.  This  finding  supports  the  notion  that  introducing  automation  in 
ATC  may  remove  the  controller  from  the  control  loop  (Endsley  &  Kaber,  1999)  and  lead  to 
decrements  in  a  controller’s  SA.  Although  the  aggregate  measure  of  Level  3  SA  proved  to 
be  sensitive  to  the  automation  manipulation,  surprisingly  there  was  no  significant  effect  of 
the  mode  of  automation  on  the  percent  correct  responses  to  individual  questions.  One 
possible  explanation  for  the  lack  of  individual  SA  question  significance  may  be  related  to  the 
level  of  workload  assigned  to  the  experiment  trials.  It  is  possible  that  7  aircraft  running  2- 
times  faster  than  real  time  was  not  high  enough  of  a  workload  to  reveal  significantly  different 
SA  across  modes  of  automation.  Further  research  is  needed  to  determine  appropriate  levels 
of  Multitask©  workload  in  order  for  the  SAGAT  to  be  a  sensitive  and  reliable  indicator  of 
the  various  modes  of  automation. 

Effects  on  Aggregate  Measures  and  Individual  Queries 

The  limited  number  of  significant  S  A  findings  in  this  research  may  also  be  attributed 
to  the  unique  ATC  environment  simulated  in  this  study  (Multitask©).  The  ATC  environment 
presents  controllers  with  a  vast  amount  of  information.  A  load  of  seven  aircraft  was  chosen 
as  the  minute-to-minute  workload  for  this  study,  considering  the  working  memory  capacity 
defined  by  Miller  (1956).  However,  the  subjects  were  required  to  recall  numerous  flight 
parameters  for  each  aircraft  during  SAGAT  queries.  Even  when  chunking  is  employed  in 
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order  to  retain  all  of  the  aircraft  parameters,  the  amount  of  information  available  within  the 
environment  may  have  exceeded  the  subjects’  working  memory  capabilities. 

Hauss  and  Eyferth  (2003)  suggested  that  controllere  may  use  an  event  based  mental 
representation  of  the  air  trafSc  situation  in  order  to  determine  what  information  is  currently 
relevant,  what  information  will  be  relevant  in  the  fiiture,  and  what  information  can  be 
neglected;  thereby  making  the  task  manageable  from  a  working  memory  perspective.  In 
addition,  Hauss  and  Eyferth  (2003)  state  that  aircraft  which  have  recently  been  contacted  by 
a  controller  and  that  have  required  recent  control  actions,  or  aircraft  that  are  currently  (or  will 
soon  be)  in  conflict,  may  demand  more  attentional  resources  than  other  display  aircraft. 
Consequently,  controllers  may  focus  on  certain  aircraft  to  the  exclusion  of  others  at  various 
times  during  control  activities.  Therefore,  when  using  SAGAT  to  assess  SA  in  ATC, 
controllers  may  be  more  or  less  prepared  to  respond  to  certain  queries  on  certain  aircraft. 
Hauss  and  Eyferth  (2003)  resolved  that  controllers  will  recall  the  flight  parameters  of  aircraft 
most  relevant  to  current  task  performance  more  accurately  in  responding  to  SAGAT  queries 
than  the  parameters  for  aircraft  that  are  not  critical  to  ATC  at  the  time  of  the  SAGAT  freeze. 
Gronlund,  Ohrt,  Dougherty,  Perry,  and  Manning  (1998)  said  that,  “not  all  aircraft  are  equally 
important  to  the  controller,  and  measures  of  SA  should  not  assume  that  they  are.”  It  is 
possible  that  in  the  current  study  subjects  responded  accurately  to  SAGAT  queries  for  the 
aircraft  that  they  had  recently  dealt  with  and  poorly  for  those  aircraft  not  in  the  focus  of  their 
attention.  Since  each  SAGAT  query  was  posed  for  all  7  aircraft  on  the  display  at  a  freeze, 
from  all  data  analyses  perspective  the  average  percent  correct  responses  to  a  single  query 
could  have  remained  relatively  constant  across  the  experimental  conditions  as  a  result  of  this 
aircraft  relevance  issue. 
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Beyond  this,  Hauss  and  Eyferth  (2003)  state  that  expert  controllers  actually  remember 
less  about  the  ATC  environment  compared  to  novice  controllers,  because  expert  controllers, 
based  on  experience,  know  what  aircraft  are  relevant  and  demand  a  heightened  amount  of 
attention.  Niessen  et  al.  (1999)  agree  that  experience  plays  a  large  roll  in  controller  SA,  as 
experienced  operators  “use  less  information,  but  that  this  information  is  diagnostically  more 
relevant”  (p.  1513).  The  subjects  in  this  study  were  not  expert  controllers,  but  they  did 
receive  3.5  hours  of  training  on  the  Multitask©  simulation.  Based  on  the  residual  and  trial 
number  plots,  no  learning  was  found,  suggesting  that  the  subjects  did  achieve  some  level  of 
expertise  in  this  ATC  simulation.  Therefore,  the  subjects  may  have  remembered  only  what 
information  was  currently  relevant  to  their  clearance  amendments.  Using  a  SA  measure  that 
placed  equal  emphasis  on  aircraft  they  considered  irrelevant  at  certain  points  in  task 
performance  may  have  been  an  insensitive  measurement  approach  for  the  domain.  Based  on 
similar  concerns,  Hauss  and  Eyferth  (2003)  actually  presented  a  method  for  SA  assessment 
which  weights  aircraft  based  on  their  relevance  to  the  current  control  scenario. 

Finally,  Hauss  and  Eyferth  (2003)  comment  that,  “a  test  is  considered  as  content  valid 
when  it  can  be  shown  that  the  questions  that  are  asked  in  the  test  are  representative  for  the 
construct  that  the  test  is  designed  for.”  The  SAGAT  queries  posed  in  this  research  were 
based  on  Endsley  and  Jones’  (1995)  GDTA  of  TRACON  and  a  GDTA  on  the  Multitask© 
simulation  (see  Appendix  A).  With  this  in  mind,  the  measurement  technique  was  considered 
to  have  good  content  validity.  In  fact,  Endsley  and  Kiris  (1995)  and  Endsley  et  al.  (2000) 
have  also  developed  SAGAT-based  measures  of  SA  in  ATC  and  have  demonstrated  content 
validity  through  empirical  research  and  association  of  S  A  results  with  performance 
outcomes.  Hauss  and  Eyferth  (2003)  go  on  to  say  that,  “any  method  that  uses  an  unweighted 
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reproduction  performance  measure  is  not  content  valid. . .  and  cannot  be  considered  as  a  valid 
method  [in  ATC]”  (p.  422). 
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7  Conclusions 


7.1  Caveats 

Multitask©  is  a  low-fidelity  ATC  simulation.  Although  its  functions  do  simulate 
cognitive  requirements  of  real  ATC,  it  has  several  limitations  when  compared  to  true  ATC 
displays  and  functions.  First,  Multitask©  displays  aircraft  in  a  2-dimensional  environment 
and  the  simulation  does  not  require  normal  3-dimensional  operations  (i.e.,  altitude 
knowledge,  vertical  separation,  vertical  approach  procedures,  etc.).  The  3-dimensional 
property  of  ATC  adds  immensely  to  the  task’s  inherent  complexity  and  increases  the  amount 
of  information  needed  by  a  controller  in  order  to  maintain  aircraft  separation.  The  vertical 
dimension  may  affect  controller  task  performance,  workload,  and  SA. 

The  Multitask©  simulation  also  lacks  other  ATC  functions  such  as  verbal 
communications,  flight  strip  management,  emergency  procedures,  and  pilot  inadvertent 
deviations  from  prescribed  routes.  These  functions  within  true  ATC  add  immensely  to  task 
workload.  In  an  attempt  to  simulate  true  ATC  workload  associated  with  these  demanding 
ATC  functions,  the  simulation  speed  within  this  study  was  increased  to  2-times  the  real  ATC 
speed.  As  mentioned  previously,  this  increase  in  simulation  speed  may  not  have  raised  task 
workload  to  the  workload  levels  of  real  ATC  operations. 

The  Multitask©  simulation  also  does  not  evoke  in  users  the  stresses  associated  with 
clearing  aircraft  that  are  carrying  human  lives.  It  is  possible  that  these  limitations  had  a 
major  impact  of  subject  motivation  and  concentration  in  the  simulation  trials  and, 
consequently,  the  workload,  performance,  and  SA  responses.  Furthermore,  the  limitations  of 

the  simulation  limit  the  generalizabilty  of  the  results  presented  here  to  the  real  ATC  domain. 
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Beyond  the  limitations  of  the  study,  based  on  the  characteristics  of  the  task,  novice 
operators  were  used  for  this  study.  Although  all  subjects  completed  a  very  thorough  training 
program  and  they  may  have  been  experts  at  performing  the  simulation,  they  still  did  not 
represent  certified  FAA  air  traffic  controllers.  Although  this  study  revealed  significant 
findings  on  AA  in  the  ATC  simulation  and  the  implications  for  controller  SA,  the  results  may 
not  be  generalizable  to  expert  ATC  controllers  due  to  the  use  of  a  novice  subject  population. 

Finally,  as  mentioned  previously,  an  unweighted  SA  measure  which  does  not  account 
for  the  varying  emphasis  that  expert  controllers  may  place  on  display  aircraft  may  not  be  a 
valid  measurement  technique  for  the  ATC  domain.  This  study  provides  additional  evidence 
to  support  this  notion. 

7.2  Design  Implications  for  Future  ATC  Automation 

The  results  of  this  study  may  be  applied  directly  to  the  design  of  future  ATC 
automation.  Similar  to  results  foxmd  by  Leroux  (1993),  Laois  and  Giannacourou  (1995), 
Kaber  et  al.  (2002),  and  Clamann  and  Kaber  (in  press),  this  study  determined  ATC 
automation  applied  to  low-level  sensory/response  functions  to  be  most  beneficial  when 
compared  to  automation  of  complex  cognitive  functions.  In  regards  to  ATC  automation 
design,  low-level  functions  may  include  aircraft  data  collection,  communications  with 
aircraft,  clearance  amendments,  and  hand-off  to  other  controllers.  These  types  of  ATC 
functions  may  benefit  most  fi-om  automation.  Conversely,  high-level  ATC  cognitive 
functions  including  aircraft  information  analysis,  clearance  decision  making,  emergency 
procedure  decisions,  and  the  comprehension  of  dynamic  relationships  between  aircraft  are 
not  as  well  suited  to  automation. 
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However,  when  implementing  AA  in  ATC,  the  automation  effects  may  not  be  this 
straightforward.  Significant  results  within  this  study  demonstrate  that  the  information 
analysis  mode  of  AA  may  decrease  task  performance,  but  manual  performance  within  this 
mode  of  automation  showed  significant  improvements  compared  to  the  action 
implementation  mode  of  AA.  This  suggests  that  controllers  can  benefit  fi-om  AA  of  high- 
level  cognitive  functions  similar  to  information  analysis,  if  the  control  for  this  function  is 
returned  to  the  human  periodically. 

This  study  also  revealed  that  a  controller’s  ability  to  project  the  future  status  of 
controlled  aircraft  is  directly  impacted  by  automation.  As  mentioned  above,  a  DFA 
scheduling  scheme  can  be  incorporated  into  ATC  design  in  which  automation  is  applied  to 
ATC  functions  to  decrease  workload,  but  when  the  controller  is  required  to  project  future 
aircraft  states,  manual  control  is  returned.  Future  research  in  regards  to  a  SA  trigger  for 
DFAs  within  ATC  is  addressed  in  Section  7.4. 

7.3  Situation  Awareness  in  Adaptive  Automation  of  Air  Traffic  Control 

This  study  attempted  to  define  a  measure  of  SA  in  an  ATC  simulation  that  is  a 
sensitive  and  reliable  indicator  of  automation  state  changes  as  part  of  AA.  The  SAGAT  is 
one  method  for  assessing  SA  within  ATC.  It  involves  first  developing  a  GDTA,  which 
identifies  the  goals,  sub-goals,  tasks  and  information  requirements  of  controllers  necessary  to 
effectively  separate  and  land  aircraft.  Secondly,  S  A  requirements  in  the  GDTA  are  used  to 
form  a  comprehensive  list  of  questions,  which  is  used  to  objectively  assess  a  controller’s 
current  perception,  comprehension,  and  projection  of  elements  in  the  ATC  environment 
during  simulation  freezes.  Previous  research  has  demonstrated  SAGAT  to  be  a  valid  and 
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reliable  measure  of  S  A  across  many  domains,  including  aircraft  cockpit  systems  (Endsley, 
1995b),  ATC  (Endsley  &  Kiris,  1995),  and  radar  monitors  (Bolstad  &  Endsley,  2000). 
However,  the  empirical  assessment  of  the  SAGAT  measure  developed  in  this  study,  in  order 
to  investigate  the  effects  of  AA  of  various  ATC  information  processing  functions,  did  not 
prove  to  be  sensitive  to  the  automation  manipulations.  Considering  the  recent  results  of 
Hauss  and  Eyferth  (2003),  mounting  evidence  suggests  that  there  may  be  another  necessary 
step  to  the  assessment  of  SA  in  ATC.  This  step  may  be  the  weighting  of  aircraft  parameters 
with  their  immediate  importance  to  the  subject  air  traffic  controller  based  on  current 
environmental  factors. 

A  study  similar  to  this  using  a  weighted  SA  measurement  technique  is  needed  in 
order  to  empirically  assess  the  utility  of  SA  measures  for  investigating  AA  of  various  ATC 
information  processing  functions.  The  results  of  such  a  study  could  be  used  to  determine  the 
most  sensitive  and  reliable  weighting  technique  for  discriminating  AA  conditions.  Finally,  a 
study  incorporating  such  a  weighting  technique  to  measure  SA  may  be  used  to  determine  the 
effect  of  AA  in  an  ATC  simulation  on  operator  SA.  Examples  of  potential  weighting 
techniques  are  presented  in  the  next  section. 

7.4  Future  Research 

Automation  is  cmrently  implemented  in  ATC  to  alleviate  workload  placed  on 
controllers  while  improving  ATC  performance  and  efficiency.  Likewise,  some  manual 
control  remains  necessary  to  prevent  controller  OOTLUF  and  the  negative  human 
performance  consequences  (complacency,  vigilance  decrements,  loss  of  S A).  Adaptive 
automation  has  been  proposed  as  an  alternative  to  conventional  automation  that  may  provide 
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the  benefits  of  moderating  operator  workload  and  eonsideration  of  OOTL  performance 
problems.  However,  the  precise  relationship  between  the  duration  of  DFA  and  the  number  of 
automation  or  manual  control  periods  needed  within  a  task  are  still  unknown.  Further 
research  using  various  secondary-task  criteria  as  bases  for  DFAs  as  part  of  AA  is  needed  to 
describe  the  relationships  between  various  proportions  of  manual  and  automated  control  to 
primary  task  performance  and  SA.  In  addition  to  defining  these  relationships,  an  optimum 
amount  of  automated  and  manual  control  time  during  AA  should  be  found  to  maximize  ATC 
and  SA  within  a  robust  ATC  simulation. 

Additional  research  is  also  needed  to  validate  the  weighted  SA  measurement 
technique  presented  by  Hauss  and  Eyferth  (2003)  within  the  Multitask©  context.  Two 
methods  are  applicable  for  applying  this  approach  to  Multitask©.  First,  an  expert 
experimenter  can,  based  on  the  current  display  setup,  define  which  aircraft  will  be  assessed 
during  a  SA  freeze.  These  aircraft  could  be  identified  based  on  recent  or  expected  clearance 
changes,  aircraft  currently  in  conflict,  or  objective  measures  of  aircraft  importance  using  eye¬ 
tracking  equipment  and  a  link  analysis  (i.e.,  index  of  importance).  Secondly,  the  same 
procedure  can  be  used  as  in  this  study,  but  subjects  could  be  asked  to  weight  the  importance 
of  queried  aircraft  in  responding  to  SAGAT  questionnaires.  This  weighting  value  (e.g.,  1-5) 
would  be  multiplied  by  the  S A  score  (percent  correct  responses  to  queries)  to  derive  a 
weighted  SA  score. 

Hauss  and  Eyferth  (2003)  attribute  the  need  for  a  weighted  SA  measure  to  the  unique 
complexity  of  the  ATC  environment  requiring  controllers  to  manage  a  large  amount  of 
information  using  an  event  based  mental  representation.  As  a  result,  varying  levels  of 
relevance  are  applied  to  each  aircraft  within  the  controller’s  airspace.  It  is  possible  that 
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relevance  of  stimuli  or  events  to  human  performance  in  other  domains,  such  as  driving  and 
piloting,  may  be  important  to  consider  in  developing  SA  measures  for  studying,  for  example, 
in-vehicle  highway  systems  or  cockpit  automation  effects  of  pilot  S  A.  Research  is  needed  to 
identify  other  environments  in  which  weighted  SA  measures  may  be  applicable  and  useful. 
Subsequently,  these  SA  measures  may  be  used  to  assess  the  effects  of  AA  of  various  HIP 
functions  on  SA  within  different  contexts. 

Once  the  effects  of  AA  on  SA  are  fully  understood,  future  research  is  needed  to 
develop  a  real-time  probe  measure  of  SA,  which  may  serve  as  a  basis  for  triggering  DFAs. 
This  method  for  allocating  control  between  the  operator  and  automation  may  produce 
superior  results  compared  to,  for  example,  a  workload  based  trigger  of  DFAs.  The  criterion 
for  this  type  of  AA  design  would  entail  allocating  control  to  automation  only  when  the 
controller  is  fiilly  “in-the-loop”  and  has  achieved  sufficient  SA.  However,  once  the 
controller  has  lost  SA,  manual  control  can  be  reinstated  to  increase  perception, 
comprehension,  and  projection  of  system  status. 
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Appendix  A:  Multitask©  Goal-Directed  Task  Analysis 


A  GDTA  is  accomplished  by  first  identifying  major  goals  required  for  accomplishing  the 
task.  Secondly,  major  subgoals  are  determined  that  are  essential  for  meeting  the  goals. 

Then,  major  decisions  that  are  associated  with  each  subgoal  are  developed  and  used  to 
determine  SA  requirements  for  accomplishing  the  task  (Endsley  &  Jones,  1995).  These 
requirements  focus  not  only  on  what  data  the  controller  needs,  but  also  on  how  that 
information  is  integrated  or  combined  to  address  each  decision  (Endsley  &  Rodgers,  1994,  p. 

4). 

Endsley  and  Rodgers  (1994)  identify  four  caveats  that  relate  to  a  GDTA: 

(1)  At  any  given  time,  more  than  one  goal  or  subgoal  may  be  operating,  although 
these  will  not  always  have  the  same  priority.  The  analysis  does  not  assiime  any 
prioritization  among  goals,  or  that  each  subgoal  within  a  goal  will  always  be  relevant. 

(2)  The  analysis  is  based  on  goals  or  objectives,  and  is  as  technology-free  as  possible. 
How  the  information  is  acquired  is  not  addressed. 

(3)  The  analysis  seeks  to  determine  what  controllers  would  ideally  like  to  know  to 
meet  each  goal. 

(4)  Static  knowledge,  such  as  procedures  or  rules  for  performing  tasks,  is  outside  the 
bounds  of  this  analysis. 

(Endsley  &  Rodgers,  1994,  p.  4-5) 

The  following  GDTA  describes  the  goals  and  information  requirements  required  to 
successfully  clear  and  land  aircraft  at  one  of  two  airports  in  Multitask©;  using  the  methods 
described  by  Endsley  and  Rodgers  (1994)  and  Endsley  and  Jones  (1995).  Each  major 
decision  and  SA  requirement  associated  with  a  subgoal  represents  a  level  of  information 
processing  (perception,  comprehension,  or  projection),  and  were  used  to  develop  SAGAT 
queries  (see  Appendix  B).  This  analysis  assumes  expert  performance  (i.e.,  operator  errors 
are  not  considered). 

Directly  following  the  GDTA  is  a  description  of  a  plan  for  accomplishing  the  goals  within 
the  Multitask©  GDTA.  Finally,  a  description  of  the  Multitask©  GDTA  explains  the  task 
goals  and  subgoals  in  more  detail. 

(Note:  Some  decisions  or  questions  make  reference  to  other  information/SA  requirements. 
These  requirements  are  emboldened.) 
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Goal  Subgoal  Decisions/SA  requirements 


Level  of  SA 


0.  land  aircraft  safely 

1.  acquire  aircraft  info 

1.1  locate  aircraft 

what  display  sector  is  the  aircraft  located? 
aircraft  position 

how  many  other  aircraft  are  located  in  that  sector? 
aircraft  position 
location  of  other  aircraft 


1.2  contact  aircraft 

1.3  verify 

is  the  aircraft  queried  (contacted)? 

aircraft  icon  (flashing  or  solid) 
aircraft  parameters 

1.4  acquire  information 

what  is  the  aircraft  type? 
aircraft  call  sign 
aircraft  speed 

apparent  velocity  of  aircraft 
what  is  the  aircraft  call  sign? 

aircraft  call  sign 
what  is  the  aircraft  speed? 
aircraft  speed 

apparent  velocity  of  aircraft 
what  is  the  destination  airport? 
destination  airport 
apparent  aircraft  route 
what  is  the  destination  runway? 

destination  runway 
what  is  the  aircraft  heading? 
destination  airport 
apparent  aircraft  route 
what  is  the  distance  to  destination? 
destination  airport 
aircraft  position  (1.1) 
what  is  the  distance  to  nearest  holding  fix? 
aircraft  position  (1.1) 
location  of  nearest  holding  fix 
what  is  the  heading  to  nearest  holding  fix? 
aircraft  position  (1.1) 


Level  1 
Level  1 

Level  1 

Level  1 

Level  1 
Level  1 

Level  1 

Level  1 
Level  2 

Level  2 

Level  2 

Level  2 
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location  of  nearest  holding  fix 
is  aircraft  speed  representative  of  type? 
aircraft  call  sign 
aircraft  speed 

apparent  velocity  of  aircraft 

2.  identify  potential  conflicts 

2.1  determine  current  relationships  between  aircraft 

what  is  the  distance  to  other  aircraft? 
aircraft  position  (1.1) 
location  of  other  aircraft  (1.1) 
what  is  the  heading  to  other  aircraft? 
aircraft  position  (1.1) 
location  of  other  aircraft  (1.1) 
do  aircraft  meet  or  exceed  lateral  separation? 
aircraft  position  (1.1) 
location  of  other  aircraft  (1.1) 
aircraft  destination  runway  (1.4) 
other  aircraft  destination  runways  (1.4) 

2.2  predict  future  information 

what  sector  will  aircraft  be  located  in  the  future? 
aircraft  position  (1.1) 
aircraft  destination  (1.4) 
aircraft  speed  (1.4) 
apparent  aircraft  route 
will  aircraft  overtake  each  other? 
aircraft  position  (1.1) 
aircraft  destination  (1.4) 
aircraft  speed  (1.4) 
apparent  aircraft  route 
other  aircraft  positions  (1.1) 
other  aircraft  destinations  (1.4) 
other  aircraft  speeds  (1.4) 
other  apparent  aircraft  routes 
do  aircraft  paths  cross  each  other? 
aircraft  position  (1.1) 
aircraft  destination  (1.4) 
aircraft  speed  (1.4) 
apparent  aircraft  route 
other  aircraft  positions  (1.1) 
other  aircraft  destinations  (1.4) 
other  aircraft  speeds  (1.4) 
other  apparent  aircraft  routes 
when  will  aircraft  converge? 


Level  2 

Level  2 

Level  2 

Level  2 

Level  3 

Level  3 

Level  3 

Level  3 
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aircraft  position  (1.1) 
aircraft  destination  (1.4) 
aircraft  speed  (1.4) 
apparent  aircraft  route 
other  aircraft  positions  (1.1) 
other  aircraft  destinations  (1.4) 
other  aircraft  speeds  (1.4) 
other  apparent  aircraft  routes 
where  will  aircraft  converge? 
aircraft  position  (1.1) 
aircraft  destination  (1.4) 
aircraft  speed  (1.4) 
apparent  aircraft  route 
other  aircraft  positions  (1.1) 
other  aircraft  destinations  (1.4) 
other  aircraft  speeds  (1.4) 
other  apparent  aircraft  routes 
when  will  aircraft  arrive  at  airport? 
aircraft  position  (1.1) 
destination  airport  (1.4) 
aircraft  speed  (1.4) 
apparent  aircraft  velocity 

3.  decide  which  aircraft  clearance  needed 

3.1  choose  aircraft  to  manipulate 

which  aircraft  will  need  to  be  manipulated? 
potential  conflict  (2.2) 
aircraft  capabilities 
projected  effect  on  other  aircraft 


3.2  choose  operation 

which  operation  should  be  used? 
potential  conflict  (2.2) 
operations  which  can  be  used 
aircraft  capabilities 
projected  effect  on  other  aircraft 

4.  provide  clearance  to  chosen  aircraft 

4.1  select  aircraft 

4.2  select  clearance 

4.3  submit 

4.4  verify 

is  aircraft  conforming  to  assigned  parameter(s)? 

changes  to  aircraft  parameters  (speed,  route,  etc) 
is  aircraft  conforming  fast  enough? 


Level  3 


Level  3 


Level  3 

Level  3 


Level  1 
Level  1 
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Plan  1  : 
Plan  2: 

Plan  3: 
Plan  4: 


aircraft  position  (1.1) 
aircraft  destination  (1.4) 
aircraft  speed  (1.4) 
apparent  aircraft  route 
other  aircraft  positions  (1.1) 
other  aircraft  destinations  (1.4) 
other  aircraft  speeds  (1.4) 
other  apparent  aircraft  routes 

are  other  aircraft  diverting  without  clearance?  Level  1 

changes  to  aircraft  parameters  (speed,  route,  etc) 
reason  for  non-conformance?  Level  2 

aircraft  position  (1.1) 
operations  which  can  be  used 

is  aircraft  conflict  resolved?  Level  2 

aircraft  position  (1.1) 
aircraft  destination  (1.4) 
aircraft  speed  (1.4) 
apparent  aircraft  route 
other  aircraft  positions  (1.1) 
other  aircraft  destinations  (1.4) 
other  aircraft  speeds  (1.4) 
other  apparent  aircraft  routes 

is  aircraft  resuming?  Level  1 

changes  to  aircraft  parameters  (speed,  route,  etc) 

do  1.1  - 1.2  - 1.3  -  1.4,  if  aircraft  is  still  flashing,  then  repeat 

does  2.1  and  2.2  reveal  two  aircraft  to  be  converging? 
if  yes,  then  perform  Plan  3 
if  no,  repeat  Plan  1 

do  3.1  -  3.2,  then  perform  Plan  4 

do  4.1  -  4.2  -  4.3  -  4.4,  if  aircraft  is  non-compliant,  then  repeat  Plan  2;  if 
aircraft  is  compliant  and  operation  was  ‘hold’,  repeat  Plan  2  until  there  is  no 
conflict,  then  issue  ‘resume’  clearance 
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Descriptions;  ATC  goals  in  the  context  of  Multitask 


0.  Land  Aircraft  Safely  -  The  overarching  goal  of  Multitask  is  to  land  the  aircraft  safely  at 
either  of  the  two  airports.  Actions  that  constitute  unsafe  flight  are:  (1)  potential  aircraft 
conflicts,  and  (2)  actual  aircraft  collisions.  The  task  goal  is  accomplished  by  successfully 
contacting  the  aircraft,  analyzing  each  aircraft’s  information,  and,  if  needed,  issuing 
additional  aircraft  clearances. 

1 .  Contact  aircraft  -  Locate  and  maintain  contact  with  aircraft  on  the  display. 

1.1  Locate  aircraft  -  Find  aircraft  on  the  display  and  place  cursor  on  aircraft. 

1 .2  Contact  aircraft  -  Double-click  left  mouse  button  on  top  of  aircraft.  Click  on 
‘Query’  in  the  control  box.  Wait  for  ‘Processing...’  to  finish. 

1 .3  Verify  —  Confirm  that  the  aircraft  icon  is  no  longer  flashing.  If  the  icon  is  still 
flashing,  then  repeat  Steps  1.1  and  1.2. 

1.4  Acquire  aircraft  information: 

Aircraft  type  —  Military,  commercial,  or  private. 

Aircraft  call  sign  -  Alpha-numeric  designation 
Speed  -  Dependent  on  aircraft  type 
Airport  -  A1  or  A2 

Runway  -  R1  or  R2  at  the  designated  airport 
Location  -  In  relation  to  airports  and  other  aircraft 
Heading  -  Based  on  icon  movement 

2.  Identify  potential  conflicts  -  Operators  are  required  to  determine  if  a  potential  conflict 
with  aircraft  is  likely. 

2.1  Determine  relationships  between  aircraft  and  two  possible  destinations 

2.2  Predict  future  flight  parameters  of  aircraft  and  determine  if  a  change  in  aircraft 
parameters  is  required 

3.  Decide  which  aircraft  clearance(s)  is/are  needed  -  If  a  conflict  is  determined  likely,  the 
operator  must  decide  the  correct  actions  to  take. 

3.1  Identify  aircraft  —  Which  aircraft  need(s)  to  be  changed? 

3.2  Identify  operation  —  Which  operation  should  be  used  to  resolve  conflict? 

Reduce  speed 
Hold 

Change  airport 
Change  runway 

4.  Issue  clearance  -  The  proper  operations  are  performed  to  resolve  conflict 

4.1  Select  aircraft  —  Double-click  left  mouse  button  on  top  of  appropriate  aircraft 

4.2  Select  clearance  —  Chose  the  correct  operation  from  the  control  box 

4.3  Submit  -  Press  ‘Submit’  on  the  control  box 

4.4  Verify  clearance  -  Verify  that  the  appropriate  aircraft  conformed  to  the 
assigned  parameters  (both  from  the  control  tox  script  and  the  visual  display);  if 
the  aircraft  was  issued  a  ‘hold’  clearance,  a  ‘resume’  clearance  is  required  after 
the  conflict  is  resolved 
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Appendix  B:  SAGAT  Questions 


Subjects  marked  on  the  below  display  where  each  aircraft,  1-7,  was  located  at  the  time  of  a 
simulation  freeze.  The  answers  to  the  following  questions  were  based  on  the  numbers 
assigned  to  each  aircraft.  Subjects  could  refer  back  to  this  diagram  and  a  list  of  question 
descriptions  (see  Appendix  G)  while  answering  questions.  Subjects  also  received  additional 
training  on  answering  questions  that  referred  to  aircraft  headings  (see  Level  2  SA  questions). 

Instructions:  Using  the  following  diagram,  indicate  where  the  7  aircraft  are  currently  located 
by  randomly  assigning  a  number  1-7  to  each  aircraft  and  writing  the  number  on  the  graphic. 


The  tables  below  presents  the  queries  to  be  asked  during  SAGAT  freezes.  Every  colimm 
asks  a  question  that  must  be  answered  for  each  aircraft  (1-7).  Every  space  has  only  one 
correct  answer  based  on  the  current  task  situation.  If  needed,  response  criterion  are  identified 
below  the  question.  Each  fireeze  included  a  random  selection  of  6  questions  from  the  pool  of 
18-questions. 
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Level  1  SA  Questions 


Has  the 
aircraft  been 
queried?  (Y 
or  N) 

What  is  the 

aircraft's 

type? 

(Military, 
Commercial, 
or  Private) 

D 

What  is  the 
aircraft's 

destination 
runway?  (R1 
orR2) 

O 

What  is  the 
aircraft's 

destination 
airport?  (A1 
orA2) 

CO 

Is  the  aircraft 
flying  at  its 
original  or 
reduced 

speed? 

< 

What  is  the 
aircraft's  cail 
sign? 

CM 

CO 

lO 

<o 

C 

O 

(U 

o 

Level  2  SA  Questions 


u. 

The  aircraft 
shares  its 
assigned 
route  with 
how  many 
other 

aircraft?  (0-7) 

LU 

Does  the 
aircraft  meet 
or  exceed 
lateral 
separation 
requirements 
?  (Y  or  N) 

3nm  or  20nm 
from  center 
bound  for 
same  runway 

Q 

What  is  the 
aircraft's 

heading  to 
the  nearest 
aircraft?* 
(degrees) 

within  20  deg 

O 

What  is  the 
aircraft's 

distance  to 
the  nearest 
aircraft? 

(nm) 

within  5  nm 

CD 

What  is  the 
aircraft's 

distance  to 
destination? 

(nm) 

within  5  nm 

< 

What  is  the 
aircraft's 
heading?* 
(degrees) 

within  20  deg 

CM 

CO 

<0 

i^' 

C 

o 

B 

6 

00 

O 


Level  3  SA  Questions 


u. 

Rank  the 
aircraft  in  the 
order  that  they 
will 

land/sequence. 

(1-7) 

When  will  the 
aircraft  arrive 
at  the 
destination 
airport?  (min 
from  now) 

I 

within  2  min 

D 

Without  a 
clearance 
change,  will 
the  aircraft's 
path  cross 
another 
aircraft's 
path?  (Y  or 

N) _ 

O 

Without  a 
clearance 
change,  will 
the  aircraft 
overtake  any 
other 

aircraft?  (Y  or 
N) 

CO 

if  a  clearance 
change  is 
needed, 
when  will 
the  change 
be  issued? 
(min  from 
now) 

c 

1 

CM 

c 

£ 

> 

What 

clearance(s) 
will  be 
required? 

(Hid,  Red  Sp, 
Ch  Aprt,  Ch 
Rwy,  Resm, 
None) 

Criterion: 

Appendix  C: 


Appendix  D:  Informed  Consent  Form 


Informed  Consent  Form 

Department  of  Industrial  Engineering,  North  Carolina  State  University 

SITUATION  AWARENESS  EFFECTS  OF  ADAPTIVE  AUTOMATION  OF  VARIOUS  AIR 
TRAFFIC  CONTROL  INFORMATION  PROCESSING  FUNCTIONS 

I  hereby  give  my  consent  for  voluntary  participation  in  the  research  project  titled,  “Situation  Awareness  Effects  of 
Adaptive  Automation  of  Various  Air  Traffic  Control  Information  Processing  Functions.”  I  understand  that  the 
person  responsible  for  this  project  is  Dr.  David  B.  Kaber,  who  can  be  telephoned  at  (919)  515-3086.  He,  or  his 
assistant  Chris  McClemon,  has  explained  to  me  the  study  objective  of  assessing  situation  awareness  under  various 
modes  of  automation  of  an  air  traflRc  simulation.  He,  or  his  assistant,  has  agreed  to  answer  any  inquiries  I  may 
have  concerning  the  procedures  of  fhe  research  and  has  informed  me  of  my  right  to  refuse  to  answer  any  specific 
questions  asked  of  me.  He,  or  his  assistant,  has  also  informed  me  that  I  may  contact  the  North  Carolina  State 
University  (NCSU),  Institutional  Review  Board  for  the  Protection  of  Human  Subjects  by  writing  them  in  care  of 
Dr.  Matt  Zingraff,  Chair  of  IRB,  Research  Administration,  NCSU,  1  Lea2ar  Hall,  Box  7514,  Raleigh,  NC  27695, 
or  by  calling  (919)  515-2444. 

Dr.  Kaber,  or  Mr.  McClernon,  has  explained  to  me  the  procedures  to  be  followed  in  this  study  and  the  potential 
risks  and  discomforts.  In  summary  the  procedures  include:  (1)  an  equipment  femiliarization  period;  (2)  training 
sessions  including  instructions  on  an  air  traffic  control  simulator.  Multitask©,  and  its  modes  of  operation;  (3)  a 
training  session  on  a  gauge-monitoring  task;  (4)  a  training  session  on  dual-task  performance  involving  both 
Multitask©  and  the  gauge-monitoring  task;  (5)  a  training  session  on  dual-task  performance,  including  situation 
awareness  questions;  (6)  ten  experimental  trials,  and  (7)  appropriate  breaks  throughout  the  experiment.  My 
experimental  participation  will  take  approximately  ten  hours  spanning  two  consecutive  days.  The  risks  have 
also  been  explained  to  me  as  follows:  (1)  a  potential  exists  for  soreness  of  the  hand  and  forearm  muscles  from 
extensive  use  of  a  mouse  controller,  and  a  standard  keyboard  in  performing  the  Multitask©  and  gauge-monitoring 
simulations;  and  (2)  a  potential  exists  for  visual  strain  and/or  fatigue  in  viewing  the  simulation  displays  through  a 
conventional  PC  monitor.  These  risks  are  not  substantially  different  from  those  associated  with  my  everyday  PC 
use.  In  the  event  that  I  experience  &tigue  or  discomfort,  I  will  inform  the  experimenters  immediately. 

Information  concerning  compensation  for  my  participation  in  this  study  has  been  explained  to  me  as  follows:  (1)  I 
will  receive  a  payment  of  $10.00  per  hour  for  my  participation  in  flie  ejqjeriment.  The  experiment  is  expected  to 
last  for  approximately  ten  hours  (approximately  $100  total).  (2)  I  will  be  entered  in  a  contest  to  receive  a  $30 
restaurant  gift  certificate  based  on  my  combined  performance  in  the  Multitask©  and  the  gauge-monitoring  task. 

(3)  In  the  event  that  I  choose  to  terminate  my  participation  in  the  experiment,  I  will  be  paid  for  only  the  time  I  have 
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provided  at  a  rate  of  $10.00  per  hour.  (4)  The  researchers  for  the  study  have  the  right  to  terminate  mv  participation 
if  I  am  not  cooperative  or  I  experience  discomfort  or  fatigue. 

I  understand  that  if  this  research  project  results  in  any  physical  or  mental  harm  to  me,  treatment  is  not 
necessarily  available  at  the  NCSU,  Student  Health  Services,  nor  is  there  necessarily  any  insurance  carried  by  the 
University  or  its  personnel  applicable  to  cover  any  such  injury.  Further  information  about  these  matters  may  be 
obtained  from  the  Institutional  Review  Board  at  (919)  515-2444. 1  Leazar  Hall  NCSU  Campus. 

I  understand  that  I  will  not  derive  any  therapeutic  treatment  from  participation  in  this  study.  I 
understand  that  I  may  discontinue  participation  in  this  study  at  any  time  without  prejudice.  I 
understand  that  all  data  will  be  kept  confidential  and  that  my  name  will  not  be  used  in  any  reports, 
written  or  unwritten. 


Signature  of  Subject: 


Date: 


Signature  of  Authorized  Representative: 


(Chris  McClemon) 
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Appendix  E:  Payment  Form 


PAYMENT  FORM  FOR  SUBJECT 

Situation  Awareness  in  ATC  Adaptive  Automation 


Name: 

Address: 


Email: 

Phone: 


Social  Security  Number: 


Day  1 

Subject 

Initials 

Day  2 

Subject 

Initials 

Days 

Subject 

Initials 

Date  (XX/XX/03) 

Start  Time  (XXXXam/pm) 

Finfeh  Time  (XXfXXanVpm) 

Total  Time  (XXXmin) 

Time  x  $.1666  ($  10/hr) 

Total  Due  (Day  1  +  Day  2  +  Day  3)  =  $ 


I  verify  that  the  above  information  is  correct, 


Signature  of  Subject  Chris  McClemon 

Ergonomics  Lab,  NCSU  IE 


Please  take  this  form  to  Mrs.  Sankaran  in  the  I£  Department,  Riddick  room  331c. 
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Appendix  F:  Contest  Rules  and  Score  Sheet 


Situation  Awareness  and  Adaptive  Automation 

Contest  Rules  and  Score  Sheet 

This  form  describes  the  Situation  Awareness  (SA)  and  Adaptive  Automation  contest  rules,  indicates  your  intent 
to  participate,  and  serves  as  the  verification  of  your  score.  If  you  are  participating  in  the  contest,  you  must  keep 
this  form  until  the  results  of  the  contest  are  reported. 

Prize:  This  contest  will  have  one  winner,  who  will  receive  a  $30,00  gift  certificate  to  the  42"^  Street  Oyster 
Bar,  located  at  508  West  Jones  Street,  Raleigh  NC  (919.831.281 1). 

Scoring:  Score  will  be  calculated  according  to  the  following  formula: 

Total  score  =  [(x  +  y  +  z)/a]  +  g 

Cleared  aircraft  (x)  =  +1 
Conflicts  (y)  =  -.5 
Collisions  (z)  -1 

Amendments  (a)  =  sum  of  clearance  changes 
Gauge-monitoring  errors  (g)  =  -1 

This  formula  will  be  applied  to  the  data  collected  during  your  performance  in  all  10  experiment  trials  (30 
minutes  each)  to  produce  a  combined  total. 

Redeeming  your  prize:  The  winner  will  be  contacted  within  one  week  after  the  last  experimental  trial  (Middle 
of  September;  make  certain  that  you  provided  your  correct  contact  information  on  the  Subject  Info  database). 

All  questions  should  be  directed  to  Chris  McCiemon  at  mcclemonck@hotmail.com 


[  ]  I  have  read  and  understand  the  contest  mles,  and  I  wish  to  participate. 
[]  I  do  not  wish  to  participate. 


Participant  Signature  Date  Chris  McCiemon 


Subject  Number 

Score 
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Appendix  G:  SAGAT  Question  Descriptions 


1.  What  is  the  aircraft’s  call  sign? 

Provide  both  the  letters  and  numbers  that  make  up  the  aircraft’s  call  sign  (ex. 

NC3999,  M8,  GA9,  etc.) 

2.  Is  the  aircraft  flying  at  its  original  or  reduced  speed?  (original  or  reduced) 

Determine  if  the  aircraft  is  flying  at  its  originally  assigned  speed,  or  if  a  ‘Reduce 
Speed’  command  has  been  issued. 

3.  What  is  the  aircraft’s  destination  airport?  (A1  or  A2) 

Identify  the  airport  the  aircraft  is  currently  assigned  to  land  at. 

4.  What  is  the  aircraft’s  destination  runway?  (R1  or  R2) 

Identify  the  runway  the  aircraft  is  currently  assigned  to  land  on. 

5.  What  is  the  aircraft  type?  (Military,  Commercial,  or  Private) 

Based  on  the  call  sign  and  speed:  military  aircraft  are  the  fastest  and  are  denoted  by 
the  letter  ‘M’  followed  by  a  sequential  number;  commercial  aircraft  fly  at  intermediate 
speeds  and  are  denoted  by  two  random  letters  followed  by  4  random  numbers;  private  aircraft 
are  the  slowest  and  are  denoted  by  ‘GA’  followed  by  a  sequential  number. 

6.  Has  the  aircraft  been  queried?  (Yes  or  No) 

Determine  if  the  aircraft  has  been  contacted,  or  is  it  still  flashing.  Aircraft  currently 
being  processed  (‘Processing...’)  are  not  yet  queried. 

7.  What  is  the  aircraft’s  heading?  (degrees) 

Using  the  compass  diagram,  determine  in  what  direction  the  aircraft  is  traveling 
towards  its  destination,  whether  it  be  an  airport  or  a  holding  fix.  If  currently  at  a  holding  fix, 
the  heading  should  read  zero. 

8.  What  is  the  aircraft’s  distance  to  destination?  (nm) 

In  nautical  miles,  determine  how  far  the  aircraft  is  from  its  destination,  whether  it  be 
an  airport  or  a  holding  fix.  If  currently  at  a  holding  fix,  the  distance  should  be  zero. 

9.  What  is  the  aircraft’s  distance  to  the  nearest  aircraft?  (nm) 

In  nautical  miles,  determine  how  far  the  aircraft  is  from  the  nearest  aircraft. 

10.  What  is  the  aircraft’s  heading  to  the  nearest  aircraft?  (nm) 

Using  the  compass  diagram,  determine  in  what  direction  the  nearest  aircraft  is. 

1 1 .  Does  the  aircraft  meet  or  exceed  lateral  separation  requirements?  (Yes  or  No) 
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An  aircraft  is  not  meeting  minimum  separation  requirements  if:  (a)  the  aircraft  is 
within  3nm  of  another  aircraft,  or  (b)  two  aircraft  are  within  20nm  of  the  radarscope  center 
and  are  bound  for  the  same  runway  at  the  same  airport. 

12.  The  aircraft  shares  its  assigned  route  with  how  many  other  aircraft?  (0-7) 

A  route  is  a  straight  line  connecting  the  aircraft’s  current  starting  point  and 
destination  (i.e.,  a  starting  location  and  holding  fix,  a  holding  fix  and  airport,  or  a  starting 
location  and  airport).  If  two  aircraft  are  flying  the  same  direction  on  the  same  route,  no 
matter  how  far  apart  they  are,  they  are  sharing  a  route. 

13.  What  clearance  will  be  required? 

If  an  aircraft  is  at  a  holding  fix,  it  must  receive  either  a  ‘Resume’  or  ‘Change  Airport’ 
command  to  continue  towards  its  destination.  Aircraft  that  may  collide  in  the  future  may  also 
require  clearance  changes.  Aircraft  outside  the  holding  fixes  may  only  be  issued  a  ‘Reduce 
Speed’  and  ‘Hold’  command.  Aircraft  inside  30nm  on  the  radarscope  may  only  be  issued  a 
‘Reduce  Speed’  and  ‘Change  Runway’  command. 

14.  If  a  clearance  change  is  needed,  when  will  the  change  be  issued?  (min.  from  now) 

Based  on  the  current  status,  determine  (in  minutes  from  the  current  time)  how  soon  a 
change  will  be  issued.  If  no  change  is  required,  answer  ‘NA’. 

15.  Without  a  clearance  change,  will  the  aircraft  overtake  any  other  aircraft?  (Y es  or  No) 

Determine  if  the  aircraft  shares  its  route  with  any  other  aircraft,  and  if  the  two  aircraft 
will  collide  as  a  result  of  one  aircraft  flying  past  the  other  aircraft. 

16.  Without  a  clearance  change,  will  the  aircraft’s  path  cross  another  aircraft’s  path?  (Yes  or 
No) 

Based  on  the  current  status,  determine  if  any  of  the  aircraft’s  current  routes  cross  each 
other,  regardless  of  lateral  separation. 

17.  When  will  the  aircraft  arrive  at  the  destination  airport?  (min.  from  now) 

Determine  when  an  aircraft  will  land. 

18.  Rank  the  aircraft  in  the  order  that  they  will  land.  (1-7) 

Determine  when  an  aircraft  will  land  in  relation  to  the  other  aircraft. 
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Appendix  H:  Closing  Questionnaire 


What  was  your  strategy  for  clearing  targets  and  controlling  the  gauge? 


Did  you  think  it  was  easier  or  more  difficult  to  use  the  automation? 


Was  it  easier  to  answer  SA  questions  while  the  simulation  was  operating  under  automation  or 
manual  control? 


Which  form  of  automation  was  most  useful? 


Are  there  any  improvements  that  can  be  made  to  the  training? 


Are  there  any  improvements  that  can  be  made  to  the  interface  or  to  the  controls  of  either 
task? 
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Appendix  I:  Subject  Instructions 


Day  1 

I.  Introduction 

[Add  “Date”  and  “Start  Time”  to  the  front  cover  of  the  subject’s  folder.  Show  subjects 
where  the  bathroom,  restroom,  drinks,  snacks,  and  personal  storage  space  on  shelf  is. 
Explain  that  subjects  may  only  have  a  snack  or  drink  during  one  of  the  breaks.  Ask 
subjects  to  turn  their  cell  phone  off.  Place  the  “Experiment  in  Progress”  sign  on  the 
door.] 

Welcome  and  thank  you  again  for  volunteering  to  participate  in  this  experiment.  This 
experiment  is  funded  by  NASA  and  its  results  will  help  design  better  air  traffic  control,  or 
ATC,  systems  in  the  future.  This  experiment  is  being  conducted  to  study  the  effects  of 
automation  on  operator  situation  awareness  in  an  air  traffic  control  simulator  and  a  gauge¬ 
monitoring  task.  The  ATC  task  will  require  you  to  use  a  standard  PC  mouse  to  safely  guide 
aircraft  in  landing  at  airports.  The  gauge-monitoring  task  will  require  you  to  use  a  standard 
PC  keyboard  to  maintain  a  moveable  pointer  within  an  ‘acceptable’  range  on  a  fixed  scale. 
Different  levels  and  durations  of  computer  assistance  will  be  provided  in  performing  the 
tasks.  In  addition,  a  series  of  questionnaires  will  be  administered  during  freezes  in  the 
simulation  to  test  your  knowledge  of  current  and  future  states  of  the  ATC  task,  or  assess  your 
situation  awareness. 

The  procedures  to  be  followed  include  [present  the  experiment  schedule  to  subject]: 
an  equipment  familiarization  period; 
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a  IS-minute  training  session,  during  which  you  will  be  provided  with  instructions  on  manual 
radar  monitoring; 

a  5-minute  training  session  on  the  gauge-monitoring  task; 

a  60-minute  training  session  on  four  different  modes  of  automation  of  the  ATC  simulation; 
a  15-minute  training  session  on  adaptive  automation  of  the  task; 
a  25-minute  practice  session  involving  both  the  radar  monitoring  and  gauge-monitoring 
tasks; 

a  10-minute  situation  awareness  questionnaire  familiarization;  and 
a  30-minute  practice  session  requiring  you  to  perform  the  radar  monitoring  and  gauge¬ 
monitoring  tasks,  and  to  complete  the  SA  questionnaires  (this  practice  will  be  representative 
of  the  experiment  trials). 

These  steps  will  all  be  conducted  on  the  first  day  of  the  experiment.  On  Day  2  and  Day  3  of 

the  experiment,  you  will  perform  the  following: 

a  15-minute  simulation  review  at  the  beginning  of  each  day; 

ten,  30-min  test  trials  including  manual  control  and  automation  of  the  ATC  task;  and 

a  debriefing  at  the  conclusion  of  each  day. 

The  total  expected  duration  of  the  experiment  over  the  three  day  period  is  approximately  13 
hours.  You  will  be  provided  with  breaks  at  least  every  90-minutes  during  your  sessions. 
You  are  asked  to  complete  all  experimental  testing;  however,  you  may  discontinue  your 
participation  at  any  time  and  you  will  be  paid  for  any  amount  of  time  you  participated.  If 
you  withdraw  from  the  experiment  without  completing  all  of  the  test  trials,  you  will  not  be 
eligible  for  winning  the  contest  gift  certificate. 
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All  experiment  instractions  will  be  presented  to  you  verbally.  If  you  do  not  understand 
certain  instructions,  you  will  be  able  to  ask  questions  during  each  training  phase.  You  must 
carefully  follow  all  instructions  given  before  and  during  the  experiment.  If  you  fail  to  follow 
instructions  or  the  computer  equipment  malfimctions,  I  will  stop  testing.  In  these  cases,  you 
will  be  allowed  to  read  the  task  instructions  and  ask  any  questions  you  may  have,  or  I  will 
correct  the  system  malfunction.  Once  your  full  comprehension  of  the  instructions  is  ensured, 
and  the  equipment  is  in  working  order,  testing  will  re-commence.  Do  you  have  any 
questions  about  the  experiment  procedures? 

Subject  Info 

[Have  subjects  sit  in  front  of  secondary  computer.  Open  the  SAGAT  application,  select 
“Subject  Info”,  and  enter  subject  ID  number.] 

You  will  first  be  asked  to  enter  your  information  into  a  database.  Your  Subject  ID  has 
already  been  entered.  Please  fill  in  the  rest  of  this  form.  When  you  are  done,  select  “Return 
to  Main  Menu.” 

Subject  Survey 

Next  you  will  complete  a  Subject  Survey.  Please  click  on  “Subject  Survey”  on  the  computer 
interface.  This  form  asks  you  questions  about  yourself  and  verifies  your  qualifications  for 
the  study.  Please  take  a  few  moments  to  complete  the  survey.  If  you  have  any  questions,  I 
will  be  happy  to  address  them. 

[Verify  subject  is  entering  valid  data  for  participation.] 
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Informed  Consent  Form 


[Give  the  subject  the  informed  consent  form.  Summarize  the  form  and  encourage  the 
subject  to  read  the  form.] 

Next  you  will  review  and  sign  an  Informed  Consent  Form.  This  form  summarizes  the 
information  that  has  been  presented  to  you  thus  far  and  identifies  the  persons  responsible  for 
the  study.  The  form  also  addresses  North  Carolina  State  University’s  liability  in  conducting 
this  experiment.  I  encourage  you  to  read  the  form.  In  order  to  participate  in  this  study,  you 
must  have  at  least  20/20  corrected  vision  and  some  computer  experience.  Extensive  flying  or 
ATC  experience  may  also  prevent  you  from  participating  in  this  experiment.  Potential  risks 
include:  (1)  soreness  of  the  hand  and  forearm  muscles  from  extensive  use  of  a  mouse  and 
keyboard;  and  (2)  visual  strain  and/or  fatigue  in  viewing  the  simulation  displays  through  a 
conventional  PC  monitor.  These  risks  are  not  substantially  different  from  those  associated  with 
everyday  PC  use.  In  the  event  that  you  experience  fatigue  or  discomfort,  please  inform  me 
immediately. 

Please  sign  and  date  both  copies  of  the  form. 

[WhUe  the  subject  is  reviewing  and  signing  the  forms;  select  “Print  Payment  Form”  on 
the  SAGAT  application,  verify  the  subject  information,  and  print.] 

[Give  one  copy  to  the  subject  to  keep.] 

Payment  Form 

[Record  the  date  and  time  the  subject  started  participation  from  the  front  cover  of  the 
subject’s  folder.  Have  subjects  verify  their  information  on  the  payment  form.] 
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This  is  the  payment  fonn  that  will  he  used  to  calculate  your  compensation  for  participating  in 
the  experiment.  Please  verify  that  the  information  you  provided  is  accurate.  [Wait.]  Verify 
that  I  have  filled  in  the  correct  start  date  and  time,  and  initial  by  those  blocks.  I  will 
complete  this  form  for  you  during  the  three  days  of  the  experiment,  and  I  will  ask  you  to 
initial  by  all  of  the  blocks  as  I  fill  them  in.  We  will  both  sign  the  form  on  the  last  day.  It  is 
imperative  to  confirm  the  correct  start  and  stop  times  for  each  testing  day  on  this  form,  as  it 
will  determine  how  much  you  are  paid. 

Contest  Rules 

[Present  Contest  Rules  and  Score  Sheet  to  the  subject.] 

Buy  participating  in  this  experiment,  you  are  eligible  to  win  a  prize.  The  participant  who 
attains  the  highest  combined  score  in  both  the  simulation  and  the  gauge-monitoring  task 
during  all  10  experiment  trials  is  eligible  for  a  $30  gift  certificate  to  the  42"**  Street  Oyster 
Bar  and  Restaurant  in  downtown  Raleigh. 

The  information  on  this  form  includes  the  rules  of  the  contest,  the  equation  used  to  compute 
your  score,  the  instructions  for  redeeming  the  prize,  and  also  serves  as  a  verification  of  your 
score.  In  order  to  redeem  your  prize,  you  must  present  this  form  to  the  experimenter.  If  you 
wish  to  participate,  please  sign  and  date  the  form  at  this  time.  I  will  enter  your  score  in  the 
blank  at  the  bottom  when  you  complete  the  experiment  trials.  If  you  have  any  questions 
about  the  contest,  I  can  answer  them  now. 
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That  concludes  the  administrative  portion  of  the  experiment.  Do  you  have  any  questions 
about  your  participation  in  the  study? 

II.  Familiarization 

Now  I  will  familiarize  you  with  the  testing  equipment.  [Direct  subjects  to  sit  in  front  of  the 
Multitask  display.]  The  experiment  equipment  is  comprised  of  two  displays,  a  fiill-size 
keyboard,  and  a  mouse  controller,  which  have  been  set  up  here  to  resemble  a  typical  working 
scenario  in  a  ‘radar’  monitoring  operation.  The  displays  are  high-resolution  graphics 
monitors  commonly  used  with  PC-based  computing  systems.  During  training  and  actual 
testing,  the  room  will  be  darkened  to  eliminate  any  glare  reflected  off  the  surfaces  of  the 
displays.  The  keyboard  is  a  standard  124  key  unit.  You  will  only  be  required  to  use  the 
control,  shift,  and  enter  keys.  [Point  to  the  keys  that  will  be  used.]  The  mouse  is  a  simple 
controller  facilitating  movement  of  the  cursor  on  the  ‘radar’  monitor  in  both  the  horizontal 
and  vertical  directions. 

This  completes  a  brief  overview  of  the  testing  equipment  you  will  use.  Do  you  have  any 
questions  concerning  the  setup? 

III.  Multitask  Training 

[Start  Multitask  and  the  Gauge-monitoring  task.  Turn  off  gauge-monitoring  computer 
monitor.  Turn  off  overhead  lights,  and  turn  on  desk  lamp.] 

You  will  now  receive  training  in  the  MultiTask  ATC  simulator.  The  ‘radar’  monitoring  task 
involves  clearing  randomly  appearing  graphic  targets  in  the  form  of  small  white  triangles. 

The  triangles  represent  either  military,  commercial,  or  private  aircraft.  Military  aircraft  may 
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represent  a  fighter  aircraft,  like  an  F-16;  commercial  aircraft  may  represent  an  airline  aircraft, 
like  a  Delta  Boeing  737;  a  private  aircraft  may  represent  a  small  personally  owned  aircraft, 
like  a  Cessna.  Military  targets  travel  the  fastest,  followed  by  commercial,  then  private.  The 
targets  also  have  a  data  tag  affixed  to  them  which  shows  the  aircraft’s  call  sign  consisting  of 
a  letter  and  numerical  designator.  The  military  aircraft  are  designated  by  the  letter  “M”  for 
military  and  one  sequential  number.  The  private  aircraft  are  designated  by  the  letters  “GA,” 
for  “general  aircraft,”  and  one  sequential  number.  The  commercial  aircraft  are  designated  by 
a  2-letter  airline  identifier  and  a  random  4-digit  number. 

The  targets  will  follow  one  of  eight  approach  paths  [point  to  each  path]  starting  at  the 
extreme  N,  NE,  E,  SE,  S,  SW,  W,  and  NW  locations  and  proceeding  to  one  of  the  two 
airports.  The  aircraft  will  be  assigned  to  land  at  one  of  the  two  airports,  named  A1  and  A2 
[point  to  them],  and  will  be  assigned  to  one  of  the  two  airport’s  runways,  R1  or  R2. 

You  will  notice  8  small  circles  on  the  Multitask  display.  These  depict  aircraft  holding  fixes 
(or  locations  at  which  an  aircraft  may  circle  while  awaiting  clearance  to  land.)  [Point  to 
them.]  The  yellow  concentric  circles  on  the  radarscope  display  are  each  lOnm  apart.  (This 
is  very  important  for  the  situation  awareness  questions  you  will  need  to  answer  later  in  the 
experiment.) 

The  goal  of  the  task  is  to  contact  targets  before  they  reach  their  assigned  airport,  and,  if 
needed,  assign  a  clearance  change  to  avoid  conflicts  with  other  aircraft.  Do  you  have  any 
question  about  the  display? 
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[Start  MultiTask  with  ‘Trial:  Manual  Practice  (30min)’,  ‘Skill  Level:  1’,  and 
‘Automation:  None  selected*.  Have  subject  perform  each  function  as  it  is  described.] 

I  will  now  explain  each  control  function  in  Multitask.  After  explaining  each  function,  I  will 
ask  you  to  perform  that  function  on  the  simulator.  To  begin  this  training  session,  press 
“Enter”  on  the  keyboard. 

When  targets  first  appear  on  the  display,  they  are  flashing  to  indicate  they  have  not  yet  been 
contacted.  You  will  establish  a  communication  link  with  flashing  targets  by  double-clicking 
on  the  triangle  icon  with  the  left  mouse  button,  selecting  the  ‘Query’  button  in  the  command 
box  [point  to  box],  and  waiting  for  the  aircraft  information  to  appear  in  the  data  box  [point 
to  box].  Please  do  this  now.  [Wait  for  processing.]  When  there  are  multiple  aircraft,  as 
will  be  the  case  in  the  experiment  trials,  only  one  aircraft  may  be  contacted  at  a  time.  An 
aircraft  that  has  been  contacted  will  stop  flashing,  and  will  be  represented  by  a  steady,  white 
triangle  icon.  A  queried  aircraft’s  information  including  its  call  sign,  speed,  destination 
airport,  and  destination  runway  [point  to  each]  will  all  be  displayed  in  the  blue  data  box  on 
the  top  of  the  control  box. 

To  issue  a  clearance  change  to  an  aircraft  that  has  been  contacted,  you  will  double-click  on 
the  desired  target  with  the  left  mouse  button.  Then  choose  fi'om  one  of  the  five  clearance 
changes  in  the  command  box  [point  to  the  command  box]  and  select  the  ‘Submit’  button. 
The  commands  are  only  requests,  and  aircraft  may  not  conform  to  your  request.  I  will  now 
explain  each  command,  and  then  ask  you  to  perform  that  command  on  the  display  aircraft: 
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Reduce  speed  —  The  reduce  speed  command  is  used  to  slow  an  aircraft  down.  An  aircraft  can 
only  reduce  its  speed  once,  and  cannot  be  returned  to  its  previous  speed.  Please  issue  a 
reduce  speed  command  now.  Notice  that  as  you  perform  an  operation,  the  pilot/controller 
dialog  for  that  operation  appears  in  the  history  box  at  the  bottom  of  the  command  box. 

[Point  to  history  box.]  This  script  will  let  you  know  if  an  aircraft  is  conforming  to  your 
request  or  not.  The  ‘CanceT  button  may  be  used  to  interrupt  the  processing  of  a  command, 
but  once  a  command  is  given,  it  cannot  be  changed. 

Hold  —  The  hold  command  is  used  to  request  an  aircraft  to  fly  directly  to  the  closest  of  one  of 
the  eight  holding  fixes  on  the  display.  Please  issue  a  hold  command  for  the  aircraft  now. 
Notice  that  when  more  than  two  verbal  exchanges  have  occurred  in  the  history  box,  the 
history  box  may  be  scrolled  with  the  arrow  buttons  to  view  previous  commands.  Please  do 
this  now.  Once  arriving  at  a  holding  fix,  the  aircraft  icon  will  turn  yellow.  An  aircraft  will 
remain  at  a  holding  fix  until  directed  to  continue  to  the  assigned  airport  and  runway  using  the 
‘Resume’  command.  [Do  not  have  them  resume  flight  yet  -  use  ‘Change  airport*.]  An 
aircraft  can  only  fly  to  a  holding  fix  if  it  is  outside  the  30nm  circle  on  the  radarscope  display. 
[Point  to  the  appropriate  ring.] 

Change  airport  -  The  change  airport  command  is  used  to  request  an  aircraft  to  fly  to  the 
airport  to  which  it  was  not  originally  assigned.  This  command  may  only  be  used  when  an 
aircraft  is  in  a  holding  pattern.  Once  a  change  airport  command  is  issued,  the  aircraft,  after  a 
short  delay,  will  proceed  to  the  new  airport  assignment.  Please  issue  a  change  airport 
command  for  the  aircraft  now.  Note  that  the  aircraft  is  now  flying  to  the  other  airport. 
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Remember  that  the  ‘Resume’  command  may  also  be  used  to  release  an  aircraft  from  a 
holding  fix,  but  it  will  then  fly  to  its  previously  assigned  airport. 

Change  runway  -  The  change  runway  command  is  used  to  request  an  aircraft  to  land  on  a 
runway  (at  its  assigned  airport)  to  which  it  was  not  originally  assigned.  This  command  may 
only  be  used  when  an  aircraft  is  in  a  holding  pattern,  or  inside  the  SOnm  circle.  Please  issue  a 
change  runway  command  to  the  aircraft  now. 

To  review  [point  to  each  area],  outside  the  30nm  circle,  only  the  reduce  speed  and  hold 
commands  may  be  given.  While  at  a  holding  fix,  only  the  resume,  change  airport,  and 
change  runway  commands  may  be  given.  Inside  the  30nm  circle,  only  the  change  runway 
and  reduce  speed  commands  may  be  given. 

[End  the  session  and  keep  the  performance  measures  window  on  the  screen.] 

Your  simulation  score  will  be  based  on  3  factors:  cleared  aircraft,  conflicts,  and  actual 
collisions. 

—  Cleared  aircraft  are  defined  as  contacted  aircraft  arriving  safely  at  an  airport. 

—  Conflicts  are  defined  as  aircraft  traveling  within  3  nm  of  other  aircraft,  or  aircraft  that  are 
within  20  nm  of  the  radarscope  center  [point  to  the  second  yellow  circle]  destined  for  the 
same  runway  at  the  same  airport. 

—  Actual  collisions  are  defined  as  aircraft  that  come  in  contact  with  each  other  or 
simultaneously  arrive  at  the  same  airport  destined  for  the  same  runway,  in  which  case 
auditory  feedback  is  provided  and  both  aircraft  are  removed  from  the  screen.  To  prevent 
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conflicts  and  actual  collisions,  you  are  advised  to  issue  changes  to  the  aircraft’s  randomly 
assigned  clearance. 

Your  selection  of  the  various  clearance  changes  and  which  aircraft  you  apply  the  changes  to 
will  be  dependent  upon  which  target  you  think  requires  the  most  immediate  attention.  All 
targets  should  be  contacted  (not  flashing)  before  they  reach  an  airport.  Your  efficiency  will 
also  be  scored  as  the  number  of  clearance  amendments  used,  and  you  should  attempt  to 
minimize  these  amendments.  Do  you  have  any  questions  about  these  performance  measures 
or  about  the  simulation  in  general? 

We  will  now  begin  the  first  of  six  training  sessions. 

I  ask  at  this  time  that  you  remove  your  watch  from  your  person  and  place  it  in  a  pocket.  Also 
ensure  that  your  cell  phone  is  turned  off.  Due  to  the  nature  of  this  experiment,  keepmg  your 
attention  focused  on  the  ATC  task  is  very  important. 

[Select  ‘Trial:  Manual  Practice  (30min)%  ‘Skill  Level:  5’,  and  ‘Automation:  None 
selected’.  Enter  subject  number,  random  seed,  and  a  trial  number.  Do  not  check  the 
box  for  ‘Run  Gauge.’] 

You  will  have  10  minutes  to  practice  contacting  aircraft  and  resolving  conflicts  on  the  ‘radar’ 
scope.  You  should  take  advantage  of  this  practice  period  to  formulate  a  conscious  strategy 
for  effectively  clearing  targets.  You  may  take  time  to  learn  the  controls  and  get  a  feel  for  the 
interface,  as  your  performance  in  this  training  period  will  not  be  recorded. 
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Feel  free  to  move  the  keyboard,  wrist  pad,  mouse,  and  chair  around  to  a  comfortable  position 
during  training.  If  you  encoxmter  a  difficulty  during  the  session  please  do  not  hesitate  to 
bring  it  to  my  attention,  and  I  will  immediately  assist  you. 

Please  commence  training  by  depressing  the  ‘Enter’  key  on  my  mark. 

[Stop  trial  after  10-minutes.  Once  training  is  complete,  offer  an  optional  2-min  rest 
period.] 

If  you  need  to  use  the  restroom  or  get  a  drink,  feel  free  to  take  a  2-minute  break  at  this  time. 
Do  you  have  any  questions  before  beginning  the  next  training  sessions? 

IV.  Gauge-Monitor  Training 

You  will  now  be  provided  training  in  the  gauge-monitoring  task. 

[Turn  the  gauge-monitoring  computer  monitor  on.  Point  to  all  of  the  ranges  as  they  are 
described.] 

The  gauge-monitoring  task  involves  maintaining  a  moving,  white  pointer  within  an 
‘acceptable’  green  and  blue  range  on  a  fixed  scale  display.  The  pointer  moves  “up”  and 
“down”  the  scale  in  a  random  manner.  You  will  use  the  ‘Shift’  key  to  move  the  pointer  back 
up  into  the  acceptable  range  if  it  drifts  into  the  lower  red,  ‘unacceptable’  range;  or  the  ‘Ctrl’ 
key  to  move  the  pointer  back  down  into  the  acceptable  range  if  it  drifts  into  the  upper  red 
range.  [Point  to  the  keys.]  Note  that  the  ‘Control’  and  ‘Shift’  control  keys  do  not  function 
when  the  pointer  is  within  the  acceptable  green  and  blue  ranges. 

A  ‘Correct  Detection’  is  recorded  by  the  system  when  you  redirect  the  pointer  after  it  has 
drifted  into  the  ‘xmacceptable’  range.  A  ‘False  Detection’  is  recorded  when  you  attempt  to 
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redirect  the  pointer  when  it  is  in  the  ‘acceptable’  range.  This  type  of  action  is  counted 
against  you.  A  ‘Miss’  is  recorded  when  the  pointer  drifts  into  the  ‘unacceptable’  range 
without  a  control  action,  and  will  also  be  counted  against  you.  Do  you  have  any  questions 
about  the  gauge-monitoring  task?  [Select  ‘Trial:  Manual  Practice  (30min)%  any  skill 
level,  and  any  automation.  Make  sure  “Run  Gauge”  is  checked.  Turn  the  radarscope 
monitor  off.] 

You  will  now  have  five-minutes  to  practice  maintaining  the  moving  pointer  within  the 
acceptable  range  on  the  fixed  scale  display.  Your  performance  during  this  trial  will  not  be 
recorded,  so  please  feel  free  to  ask  any  questions  you  may  have  during  the  trial. 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark. 

[Stop  trial  after  5-minutes.  Once  training  is  complete,  offer  an  optional  2-min  rest 
period.] 

If  you  need  to  use  the  restroom  or  get  a  drink,  feel  free  to  take  a  2-minute  break  at  this  time. 
Do  you  have  any  questions  before  beginning  the  next  training  sessions? 

V.  Automation  training 

We  will  now  begin  the  automation  training  sessions.  The  MultiTask  simulation  is  capable  of 
operating  under  four  different  modes  of  automation  in  addition  to  the  manual  control  mode 
you  have  already  learned.  Reference  sheets  for  each  mode  of  automation  will  be  left  next  to 
the  computer  throughout  the  training  sessions  if  you  need  to  review  their  functions  at  any 
time.  You  will  also  be  allowed  to  reference  these  sheets  before  each  test  session,  but  they 
will  not  be  available  during  test  sessions.  [Show  them  the  sheets  on  the  desk.] 
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Information  Acquisition 

The  first  mode  is  called  information  acquisition.  [Show  the  subject  this  page  of  the 
reference  sheets.]  This  form  of  automation  incorporates  a  scan  line  that  rotates  cloclcwise 
around  the  radar  display,  and  as  it  passes  over  an  aircraft  icon,  a  Trajectory  Projection  Aid 
(or  ‘TPA’)  for  that  aircraft  is  presented  for  two  seconds.  The  TPA  shows  the  aircraft  route  in 
the  form  of  a  line  connecting  the  aircraft  and  the  destination  —  either  an  airport  or  holding 
fix.  The  aircraft  speed  in  knots,  destination  airport,  and  destination  runway  are  all  affixed  to 
the  center  of  the  TPA.  If  the  right  mouse  button  is  clicked  on  an  aircraft  during  the  two 
seconds  after  the  scan  line  passes  an  icon,  that  aircraft’s  TPA  will  be  displayed  for  an 
additional  8-seconds.  Do  you  have  any  questions  concerning  the  operation  of  the  simulation 
rmder  information  acquisition  automation?  [In  MultiTask  select  ‘Trial:  Automation 
Practice  (30min)’,  ‘Skill  Level:  5’,  and  ‘Automation:  Information  Acquisition’.  Enter 
subject  number,  random  seed,  and  a  trial  number.  Do  not  check  the  ‘Run  Gauge’  box.] 
You  will  now  be  given  10  minutes  to  familiarize  yourself  with  this  mode  of  automation. 

Your  performance  during  this  trial  will  not  be  recorded,  so  please  feel  free  to  ask  any 
questions  you  may  have  during  the  trial. 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark. 

[Stop  the  trial  after  10  minutes.  Once  training  is  complete,  offer  an  optional  2-min  rest 
period.] 

If  you  need  to  use  the  restroom  or  get  a  drink,  feel  fi-ee  to  take  a  2-minute  break  at  this  time. 
Do  you  have  any  questions  before  beginning  the  next  training  session? 
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Information  analysis 

The  next  automation  mode  is  information  analysis.  [Show  the  subject  this  page  of  the 
reference  sheets.]  The  information  analysis  form  of  automation  displays  information 
pertinent  to  each  of  the  contacted  aircraft  in  the  automaton  aid  box  [point  to  where  the  aid 
box  will  be  on  the  display]  including  the  aircraft’s  call  sign,  destination  airport,  destination 
runway,  speed,  and  distance  (nm)  from  the  destination  airport.  A  final  block,  ‘Conf,’  denotes 
the  call  sign  of  any  aircraft  that  are  currently  in  conflict.  The  speed  block  represents  the 
aircraft’s  speed  at  its  original  assigned  speed,  displayed  as  ‘SPDl’,  or  reduced  speed, 
displayed  as  ‘SPD2’;  however,  the  true  aircraft  speed  can  still  be  obtained  fi-om  the  data  box 
by  querying  the  aircraft.  [Point  to  the  data  box  again.]  Do  you  have  any  questions 
concerning  the  operation  of  the  simulation  under  information  analysis  automation? 

[In  MultiTask  select  ‘Trial:  Automation  Practice’,  ‘Skill  Level:  S’,  and  ‘Automation: 
Information  Analysis’.  Enter  subject  number,  random  seed,  and  a  trial  number.  Do 
not  check  the  ‘Run  Gauge’  box.] 

You  will  now  be  given  10  minutes  to  familiarize  yourself  with  this  mode  of  automation. 

Your  performance  during  this  trial  will  not  be  recorded,  so  please  feel  free  to  ask  any 
questions  you  may  have  during  the  trial. 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark. 

[Stop  trial  after  10-minutes.  Once  training  is  complete,  offer  an  optional  2-min  rest 
period.] 

If  you  need  to  use  the  restroom  or  get  a  drink,  feel  free  to  take  a  2-minute  break  at  this  time. 
Do  you  have  any  questions  before  beginning  the  next  training  session? 
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Decision  making 

The  next  automation  mode  is  decision  making.  [Show  the  subject  this  page  of  the 
reference  sheets.]  In  addition  to  conflict  alerting,  decision  making  automation  provides 
recommendations  for  resolving  aircraft  conflicts  and  displays  these  recommendations  in  the 
automation  aid  box.  The  conflicting  aircraft  IDs,  the  recommended  change  (speed,  hold, 
resumption,  or  airport),  and  the  aircraft  to  which  the  clearance  should  be  issued  are  all 
displayed  together.  Up  to  three  recommendations  are  displayed  in  the  automation  aid  box  at 
a  time  and  they  are  listed  in  order  of  priority.  Runway  changes  are  not  prescribed  by  the 
decision  aid;  ftierefore,  you  will  need  to  closely  monitor  aircraft  within  30  nm  of  the  airports. 
Do  you  have  any  questions  concerning  the  operation  of  the  simulation  under  decision  making 
automation? 

[In  MultiTask  select  ‘Trial:  Automation  Practice’,  ‘Skill  Level:  5*,  and  ‘Automation: 
Decision  Making’.  Enter  subject  number,  random  seed,  and  a  trial  number.  Do  not 
check  the  ‘Run  Gauge’  box.  Stop  the  trial  after  10  minutes] 

You  will  now  be  given  10  minutes  to  familiarize  yourself  with  this  mode  of  automation. 

Your  performance  during  this  trial  will  not  be  recorded,  so  please  feel  free  to  ask  any 
questions  you  may  have  during  the  trial. 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark. 

[Stop  trial  after  10-minutes.  Once  training  is  complete,  offer  an  optional  2-min  rest 
period.] 

If  you  need  to  use  the  restroom  or  get  a  drink,  feel  fi'ee  to  take  a  2-minute  break  at  this  time. 
Do  you  have  any  questions  before  beginning  the  next  training  session? 
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Action  implementation 

The  next  mode  of  automation  is  action  implementation.  [Show  the  subject  this  page  of  the 
reference  sheets.]  The  action  implementation  level  of  automation  simulates  the  hand-off  of 
ATC  from  approach  control  (the  task  you  have  been  performing)  to  local  tower  control. 
Subsequently,  the  tower  automatically  maintains  full  control  and  responsibility  of  all  aircraft 
within  20  nm  of  the  center  of  the  radarscope.  You  are  only  responsible  for  controlling 
aircraft  from  the  periphery  of  the  radarscope  to  the  20  nm  line.  [Point  to  the  area  between 
the  peripheiy  and  the  20  nm  line.]  Automated  control,  activated  inside  this  line,  prevents 
any  conflicts  (either  conflicts  or  collisions).  The  action  implementation  automation  aid  box 
simply  tallies  the  number  of  aircraft  on  the  radar  display,  by  type.  Do  you  have  any 
questions  concerning  the  operation  of  the  simulation  under  action  implementation 
automation? 

[In  MultiTask  select  ‘Trial;  Automation  Practice’,  ‘Skill  Level;  5’,  and  ‘Automation; 
Action  Implementation’.  Enter  subject  number,  random  seed,  and  a  trial  number.  Do 
not  check  the  ‘Run  Gauge’  box.  Stop  the  trial  after  10  minutes] 

You  will  now  be  given  10  minutes  to  familiarize  yourself  with  this  mode  of  automation. 

Your  performance  during  this  trial  will  not  be  recorded,  so  please  feel  free  to  ask  any 
questions  you  may  have  during  the  trial. 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark. 

[Stop  the  trial  after  10  minutes] 

That  concludes  the  training  you  will  receive  on  the  various  functions  of  simulation 
automation.  Do  you  have  any  questions  about  any  of  the  automation  functions  of  MultiTask? 
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There  will  now  be  a  10  min  break  prior  to  additional  practice  sessions.  If  you  need  to  use  the 
restroom  or  get  a  drink,  please  do  so  at  this  time. 

[Allow  for  a  lO-min  rest  period.] 

VI.  Dual-Task  Practice 

All  experiment  trials  will  include  adaptive  forms  of  the  previously  described  modes  of 
automation.  Under  adaptive  automation,  your  performance  on  the  gauge-monitoring  task 
will  dictate  whether  the  computer  provides  you  with  assistance  in  the  ATC  simulation,  or 
whether  you  perform  the  task  manually.  If  you  perform  poorly  on  the  gauge-monitoring  task, 
automation  will  be  allocated  in  the  simulation,  and  an  ‘Automation  is  ON’  box  will  appear  in 
the  lower  left  comer  of  the  display  [point  to  the  automation  status  box  on  the  computer 
display].  Once  your  performance  of  the  gauge-monitoring  task  improves,  the  automation 
will  be  turned  off,  manual  control  will  be  returned  to  you,  and  ‘Automation  is  OFF’  will 
appear  in  the  lower  left  comer  of  the  display. 

In  this  practice  session,  you  will  perform  both  the  gauge-monitoring  task  and  the  ATC 
simulation.  As  in  the  other  practice  sessions,  your  tasks  will  be  to  contact,  clear,  and  safely 
land  all  of  the  aircraft  on  the  display  while  maintaining  the  moving  gauge  pointer  in  the 
acceptable  range  on  the  fixed  scale  display.  You  should  focus  your  attention  on  the  ATC 
simulation,  and  attend  to  the  gauge  monitor  only  to  the  extent  that  you  can.  The  practice 
session  will  last  25  minutes.  Your  performance  during  this  trial  will  be  recorded,  so  accuracy 
is  very  important. 
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[In  MultiTask  select  ‘Trial:  Dual  Task  Practice’,  ‘Skill  Level:  7’,  and  ‘Automation: 
<Depending  on  random  assignment  -  see  Freeze  Times  table>’.  Enter  subject  number, 
random  seed,  and  a  trial  number.  Make  sure  ‘Run  Gauge’  is  checked.  Enter  a 
benchmark  gauge  performance  of  ‘1’  and  a  benchmark  standard  deviation  of  ‘0’.] 

Do  you  have  any  questions  about  the  adaptive  automation  fimction  of  MultiTask? 

You  will  operate  the  simulation  under  adaptive  automation  of  the  <see  Freeze  Times  table> 
mode  of  automation.  Do  you  have  any  questions  about  his  mode  of  automation?  [If  the 
subject  does,  allow  them  to  review  the  appropriate  reference  sheet.] 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark. 

[While  the  trial  is  running,  open  the  ‘Login’  form  on  the  SAGAT  database.  Log  the 
subject  on,  and  open  SAGAT  questions  1, 7,  and  13.  Place  a  SAGAT  questionnaire  key 
on  the  desk.] 

[Stop  the  trial  after  25  minutes.  Once  training  is  complete,  offer  an  optional  2-min  rest 
period.] 

If  you  need  to  use  the  restroom  or  get  a  drink,  feel  free  to  take  a  2-minute  break  at  this  time. 
Do  you  have  any  questions  about  the  adaptive  automation  function  of  MultiTask? 

Vn.  SAGAT 

You  will  now  be  given  training  on  the  Situation  Awareness  Global  Assessment  Technique,  or 
‘SAGAT’.  The  SAGAT  involves  a  series  of  questionnaires  presented  during  the  simulation 
that  are  intended  to  assess  your  knowledge  of  the  current  and  future  simulation  status.  Each 
experiment  trial  will  include  three  stops,  or  ‘freezes’,  and  during  each  freeze  you  will 
respond  to  6  SAGAT  questions.  At  random  times  during  test  trials,  the  Multitask  simulation 
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will  freeze  and  the  display  sereen  will  be  minimized.  You  will  not  be  able  to  acquire  any 
more  information  on  the  simulation  from  the  monitors  during  a  freeze.  You  will  then  turn 
your  chair  to  the  secondary  computer  and  immediately  complete  the  SAGAT  questioimaire 
key  that  will  be  on  the  desk.  [Show  the  subject  the  questionnaire  key.]  This  is  a  form  that 
you  will  use  as  a  basis  to  answer  all  of  the  SAGAT  questions.  You  will  be  asked  to  indicate 
where  the  7  aircraft  are  currently  located  by  randomly  assigning  a  number  1-7  to  each  aircraft 
and  writing  that  niunber  at  the  aircraft’s  corresponding  location  on  the  graphic.  You  will 
then  begin  completing  the  electronic  SAGAT  forms  that  are  open  on  the  secondary  computer 
display.  This  is  what  the  forms  will  look  like.  [Point  to  the  forms  that  are  open  on  the 
computer  display.]  Each  form  will  ask  a  question  about  all  7  aircraft,  and  you  are  required 
to  provide  an  answer  for  each  aircraft  based  on  the  numbers  you  assigned  the  aircraft. 

Some  questions  will  ask  about  the  headings  of  aircraft.  These  questions  will  be  accompanied 
by  a  diagram  of  a  compass  for  your  reference.  The  up,  or  North,  direction  from  the  aircraft 
corresponds  to  360  degrees;  the  right,  or  East  direction  from  the  aircraft  refers  to  90  degrees; 
and  so  on.  [Show  subjects  on  the  display,  and  use  the  questionnaire  key  graphic  if 
needed.]  Your  answers  for  these  questions  will  be  numerical  (between  1  and  360  degrees). 
Other  questions  may  ask  you  for  speeds,  distances,  aircraft  call  signs,  and  other  information 
that  can  be  gathered  from  the  simulation  display.  If  needed,  a  criterion  will  be  presented  on 
the  bottom  of  the  form  which  identifies  how  accurate  your  answer  must  be. 

While  you  are  answering  the  questions,  I  will  reactivate  the  simulation  monitors  in  order  to 
record  the  actual  state  of  the  Multitask  simulation.  It  is  imperative  that  you  do  not  look  at  the 
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monitors  while  answering  the  questions.  If  you  do  look  at  the  monitors  during  a  SAGAT 
freeze,  this  will  be  considered  as  cheating  and  you  will  be  immediately  removed  from  the 
experiment.  Do  you  have  any  questions  about  any  of  the  SAGAT  procedures? 

This  SAGAT  training  session  involves  you  looking  at  three  example  questions  using  the 
SAGAT  application  interface.  When  you  are  done  looking  at  a  question,  press  the  “Next” 
button  to  view  the  next  question  until  there  are  no  more  questions.  Take  5-minutes  to  look 
over  the  questions,  and  ask  any  questions  you  may  have  about  the  format  and  content  of  the 
questions.  You  may  begin. 

[While  subject  is  reviewing  the  questions,  save  the  GPRAC.txt  file  to  disk.] 

[Once  training  is  complete,  require  a  5-min  rest  period.]  There  will  be  a  5  min  break 
prior  to  beginning  the  next  practice  period.  If  you  need  to  use  the  restroom  or  get  a  drink, 
please  do  so  at  this  time 

[During  the  break,  import  the  GPRAC.txt  file  into  excel.  Calculate  the  mean  and 
standard  deviation  for  the  last  20  minutes  of  the  dual  task  practice.  Record  the  values 
on  the  front  cover  of  the  subject’s  folder.] 

Vin.  Dual-Task  and  SAGAT  Practice 

You  will  now  be  provided  one  more  practice  trial,  similar  to  the  previous  dual-task  practice; 
however,  in  this  trial  you  will  experience  three  SAGAT  freezes.  As  in  the  other  practice 
sessions,  your  tasks  will  be  to  contact,  clear,  and  safely  land  all  of  the  aircraft  on  the  display 
while  maintaining  the  moving  gauge  pointer  in  the  ‘acceptable’  range  on  the  fixed  scale 
display.  Remember,  you  should  focus  your  attention  on  the  ATC  task  and  attend  to  the 
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gauge  monitor  whenever  you  can.  The  simulation  will  also  be  stopped  at  random  points  in 
time,  and  you  will  be  required  to  answer  SAGAT  questions  as  accurately  as  possible.  This 
practice  session  will  last  30  minutes.  This  session  is  representative  of  the  experiment  test 
trials  in  that  the  aircraft  speeds  will  be  realistic  given  the  distances  presented  on  the  radar 
scope.  Unless  you  have  any  questions,  we  will  begin  the  training  session.  Your  performance 
during  this  trial  will  be  recorded,  so  accuracy  is  very  important. 

[In  MultiTask  select  ‘Trial;  Experiment  Trial’,  ‘Skill  Level:  7’,  and  ‘Automation: 
<Depending  on  random  assignment  —  see  Freeze  Times  table>.  Enter  subject  number, 
random  seed,  and  a  trial  number.  Make  sure  ‘Run  Gauge’  is  checked.  Enter  the 
performance  data  calculated  during  the  break.] 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark.  [Start  the  timer  for 
the  time  listed  on  the  Freeze  Time  table.  Place  a  questionnaire  key  at  the  secondary 
desk.  When  the  timer  goes  off,  immediately  press  the  ‘Q’  and  ‘B’  keys  on  the 
keyboard.] 

Please  turn  your  chair  to  the  secondary  desk,  and  complete  the  questionnaire  key.  [While 
subjects  are  completing  the  key,  log  the  subject  into  the  SAGAT  interface  using  the 
appropriate  automation  status  -  manual  or  automation.  Select  the  random  questions 
for  that  trial.]  Now  answer  the  questions  that  are  open  on  the  SAGAT  interface.  Continue 
to  the  next  question  until  there  are  no  more  questions  left  to  answer.  Remember,  do  not  look 
at  the  simulation  display  while  answering  SAGAT  questions,  or  you  will  be  terminated  from 
the  experiment. 

[Click  on  the  Multitask  action  bar,  and  press  the  ‘V’  key  to  maximize  the  display. 
Answer  the  random  questions  on  the  SAGAT  Answer  Sheet  using  the  subject’s 
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questionnaire  key.  Press  the  ‘B’  key  again,  and  wait  for  the  subject  to  finish.  When 
subject  is  finished,  and  repositioned  at  the  primary  simulation  display,  maximize  the 
display  and  restart  the  simulation  using  the  and  ‘R’  keys.] 

[Repeat  SAGAT  procedures  for  remaining  2  stops.] 

[Stop  the  trial  after  30  minutes] 

That  concludes  your  participation  for  the  first  day  of  the  experiment.  Do  you  have  any 
questions  about  the  applications  you  have  learned  today?  [Enter  the  time  and  payment 
amount  on  the  payment  form.]  Please  look  over  the  payment  form  closely  and  initial  by 
the  ‘Finish  Time’,  ‘Total  Time’,  and  your  earned  pay  for  today’s  participation.  The  next  two 
days  of  your  participation  will  last  more  than  5  hours  each,  so  please  come  well  rested. 
[Discuss  the  starting  date  and  time  for  the  second  and  third  day  of  participation,  and 
remind  them  to  please  be  on  time.] 

[Before  the  second  day,  import  the  GTEST.txt  gauge  data  from  the  last  practice  session 
into  Excel.  Calculate  the  mean  and  standard  deviation  for  gauge  performance  during 
the  last  20  minutes.  Record  these  figures,  and  average  them  with  the  data  from  the 
other  subject  assigned  the  particular  mode  of  automation.  Use  the  average  mean 
performance  and  standard  deviation  as  a  basis  for  triggering  AA  in  all  test  trials  with 
that  mode  of  automation.] 
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Days  2  &  3 

[Welcome  the  subject  back.  Remind  the  subject  to  turn  their  cell  phone  off.  Record  the 
date  and  exact  time  in  the  ‘Date’  and  ‘Start  Time’  blocks  on  the  Payment  Form.  Ask 
the  subject  to  initial  the  two  blocks.  Place  the  “Experiment  in  Progress”  sign  on  the 
door.] 

IX.  Review 

A  brief  review  session  will  be  conducted  to  remind  you  of  the  application  functions  you  will 
use  today.  Please  follow  along  on  the  MultiTask  automation  reference  sheets. 

[In  MultiTask  select  ‘Trial:  Experiment  Trial’,  ‘Skill  Level:  7’,  and  ‘Automation: 
none’.  Enter  subject  number,  random  seed,  and  a  trial  number.  Check  the  ‘Run 
Gauge*  box.  Enter  a  benchmark  gauge  performance  of  ‘1’  and  a  benchmark  standard 
deviation  of  ‘0’.] 

Remember,  the  goal  of  the  task  is  to  contact  targets  before  they  reach  their  assigned  airport, 
and,  if  needed,  assign  a  clearance  change  to  avoid  conflicts  with  other  aircraft.  The 
following  clearance  changes  may  be  administered:  reduce  speed,  hold,  resume,  change 
runway,  and  change  airport.  Outside  the  30nm  circle,  only  the  reduce  speed  and  hold 
commands  may  be  given.  While  at  a  holding  fix,  only  the  resume,  change  airport,  and 
change  runway  commands  may  be  given.  Inside  the  30nm  circle,  only  the  change  runway 
and  reduce  speed  commands  may  be  given.  Aircraft  must  be  queried  before  reaching  an 
airport.  Performance  measures  include  cleared  aircraft,  conflicts,  actual  collisions,  and 
clearance  amendments. 
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Your  goal  in  the  gauge-monitoring  task  is  to  keep  the  moving  pointer  within  the  green  and 
blue  ‘acceptable’  range  using  the  ‘Shift’  and  ‘Ctrl’  keys.  The  ATC  simulation  display  should 
always  be  your  priority,  and  you  should  focus  your  attention  to  the  gauge-monitoring  task 
only  when  you  can.  Do  you  have  any  questions  about  the  gauge-monitoring  task? 

You  will  now  be  given  10  minutes  to  re-familiarize  yourself  with  the  simulation.  Your 
performance  during  this  trial  will  not  be  recorded,  so  please  feel  free  to  ask  any  questions 
you  may  have  during  the  trial. 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark. 

[Stop  the  trial  after  10  minutes.  Do  not  write  data  to  file.] 

Remember  that  the  simulation  can  operate  under  four  different  modes  of  automation.  [Show 
the  subject  the  automation  reference  sheets.]  These  modes  include: 

Information  Acquisition.  A  scan  line  rotates  clockwise  around  the  radar  display,  and  as  it 
passes  over  an  aircraft  icon,  a  Trajectory  Projection  Aid  (TP A)  for  that  aircraft  is  presented 
for  two  seconds.  The  TPA  shows  the  aircraft  route  in  the  form  of  a  line  connecting  the 
aircraft  and  the  destination  --  either  an  airport  or  holding  fix.  The  aircraft  speed  (in  knots), 
destination  airport,  and  destination  runway  are  affixed  to  the  center  of  the  TPA.  If  the  right 
mouse  button  is  clicked  on  the  aircraft  during  the  two  seconds  after  the  scan  line  passes  it,  the 
TPA  will  be  displayed  for  an  additional  8-seconds.  No  information  is  displayed  in  the 
automation  aid  box.  All  other  functions  of  the  simulation  operate  as  in  the  manual  control 
mode. 
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Information  Analysis.  Information  pertinent  to  each  of  the  aircraft  on  the  radarscope  is 
displayed  in  the  automaton  aid  box  including  the  aircraft’s  call  sign,  destination  airport, 
destination  runway,  speed,  and  distance  (nm)  from  the  destination  airport.  A  final  block, 
‘Conf,’  denotes  the  call  sign  of  any  aircraft  that  are  currently  in  conflict.  All  other  fimctions 
of  the  simulation  operate  as  in  the  manual  control  mode. 

Decision  Making.  In  addition  to  conflict  alerting,  recommendations  for  conflict  resolution 
are  provided  in  the  automation  aid  box.  The  identities  of  the  conflicting  aircraft,  the 
recommended  change  (speed,  hold,  resumption,  or  airport),  and  the  aircraft  to  which  the 
clearance  should  be  issued  are  all  displayed  together.  Up  to  three  recommendations  are 
displayed  in  the  automation  aid  box  at  a  time  and  they  are  listed  in  order  of  priority.  Runway 
changes  are  not  prescribed.  All  other  functions  of  the  simulation  operate  as  in  the  manual 
control  mode. 

Action  Implementation.  This  level  of  automation  simulates  the  hand-off  of  aircraft  control 
from  approach  control  to  local  tower  control.  The  tower  automatically  maintains  full  control 
and  responsibility  of  all  aircraft  within  20  nm  of  the  center  of  the  radarscope.  Automation 
within  this  area  prevents  any  conflicts  after  the  hand-off  to  tower  control  occurs.  The  action 
implementation  automation  aid  box  displays  a  tally  of  aircraft  on  the  display  by  type.  All 
other  functions  of  the  simulation  operate  as  in  the  manual  control  mode. 

Do  you  have  any  questions  about  the  ATC  simulation  functions? 
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Finally,  the  simulation  will  be  stopped  at  random  times  prompting  you  to  fill  out  the  SAGAT 
Questionnaire  Key  and  answer  the  SAGAT  questions  on  the  secondary  computer  interface. 

If  you  look  at  the  active  simulation  display  while  answering  SAGAT  questions,  you  will  be 
terminated  from  the  experiment.  Descriptions  of  the  SAGAT  questions  are  next  to  the 
computer  screen  for  you  to  review  during  SAGAT  stops. 

Do  you  have  any  questions  about  the  SAGAT  questionnaires? 

Do  you  have  any  questions  about  any  of  the  applications  you  will  use  in  the  test  sessions? 
Before  each  test  session  you  will  be  allowed  to  review  the  reference  sheets  for  that  particular 
mode  of  automation  to  be  tested,  but  you  may  not  review  the  reference  sheets  during  a  testing 
session. 

If  you  don’t  have  any  more  questions,  we  will  begin  the  testing  portion  of  the  experiment. 

X.  Testing 

Experiment  testing  will  occur  in  a  maimer  similar  to  the  last  practice  session  on  Day  1 . 
Adaptive  automation  will  be  applied  to  the  ATC  simulation,  involving  one  of  the  four  modes 
of  automation.  Each  trial  will  incorporate  one  mode  of  automation,  control  will  shift 
between  manual  and  automated  assistance  depending  upon  your  gauge  task  performance,  and 
SAGAT  freezes  will  occur.  Each  trial  will  last  30-minutes  and  you  will  be  offered  a  break 
after  every  trial.  Your  performance  during  all  of  the  trials  today  will  be  recorded,  so 
accuracy  is  very  important.  Do  you  have  any  questions  about  the  experiment  trial 
procedures? 
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Trial  <see  ‘Mode  Order’  reference  sheet> 


This  trial  will  incorporate  the  <see  ‘Mode  Order’>  mode  of  automation.  You  will  now  be 
allowed  to  review  the  functions  of  that  mode  of  automation.  [Allow  subject  to  review  the 
appropriate  automation  reference  sheet  that  is  on  the  desk  near  the  computer.  Wait. 
Subsequently,  file  it  until  the  next  test  trial.] 

This  trial  will  last  30  minutes.  If  you  encounter  a  difficulty  during  the  session  please  do  not 
hesitate  to  bring  it  to  my  attention,  and  I  will  immediately  assist  you. 

[In  MultiTask,  select  ‘Trial:  Experiment  Trial*,  ‘Skill  Level:  7*,  and  ‘Automation:  <see 
‘Mode  Order*>.  Enter  subject  number,  random  seed,  and  trial  number.  Make  sure  the 
‘Run  Gauge*  box  is  checked.  Enter  the  average  gauge  performance  and  standard 
deviation  values  captured  during  the  dual-task  with  SAGAT  session.] 

Unless  you  have  any  questions,  we  will  begin  testing. 

Please  commence  testing  by  depressing  the  ‘Enter’  key  on  my  mark.  [Start  the  timer  for 
the  time  listed  on  the  Freeze  Time  table.  Place  a  questionnaire  key  at  the  secondary 
desk.  When  the  timer  goes  off,  immediately  press  the  ‘Q*  and  ‘B*  keys  on  the 
keyboard.] 

Please  turn  your  chair  to  the  secondary  desk,  and  complete  the  questionnaire  key.  [WhOe 
subjects  are  completing  the  key,  log  the  subject  into  the  SAGAT  interface  using  the 
appropriate  automation  status  -  manual  or  automation.  Select  the  random  questions 
for  that  trial.]  Now  answer  the  questions  that  are  open  on  the  SAGAT  interface.  Continue 
to  the  next  question  until  there  are  no  more  questions  left  to  answer.  Remember,  do  not  look 
at  the  simulation  display  while  answering  SAGAT  questions,  or  you  will  be  terminated  from 


145 


the  experiment.  Descriptions  of  the  SAGAT  questions  are  next  to  the  computer  screen  for 
you  to  review  during  SAGAT  stops. 

[Click  on  the  Multitask  action  bar,  and  press  the  key  to  maximize  the  display. 
Answer  the  random  questions  on  the  SAGAT  Answer  Sheet  referencing  the  subject’s 
questionnaire  key  for  aircraft  numbers.  Press  the  key  again,  and  wait  for  the 
subject  to  finish.  When  subject  is  finished  and  repositioned  at  the  primary  simulation 
display,  maximize  the  display  and  restart  the  simulation  using  the  ‘V’  and  ‘R’  keys.] 
[Repeat  SAGAT  procedures  for  remaining  2  stops.  The  trial  will  end  after  30  minutes. 
Be  certain  to  write  data  to  file.] 

[Allow  for  a  10-minute  break.] 

There  will  now  be  a  10  min  break  prior  to  beginning  the  next  trial.  If  you  need  to  use  the 
restroom  or  get  a  drink,  please  do  so  at  this  time. 

[Repeat  procedures  for  each  trial.] 

XI.  Conclusion 

I’d  like  to  thank  you  for  your  participation.  Again,  this  research  is  funded  by  NASA  and  will 
help  in  designing  ATC  systems  in  the  future.  I  can  now  answer  any  questions  you  have 
about  the  experiment. 

Closing  Questionnaire 

[Give  the  subject  the  closing  questionnaire.]  Please  take  a  few  minutes  to  answer  these 
questions.  If  you  have  any  questions,  feel  free  to  ask. 
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[Enter  the  time  and  payment  amount  on  the  payment  form.  Sign  the  form  as  the 
experimenter.]  Please  look  over  the  payment  form  closely  and  initial  by  the  finish  time, 
total  time,  your  earned  pay  for  today’s  participation,  and  the  total  pay  earned  for  the 
experiment.  You  will  need  to  take  this  form  to  Mrs.  Sankaran  in  the  IE  department,  room 
33  Ic,  within  the  next  two  weeks.  She  will  pay  you  cash  for  your  participation.  I  will  contact 
you  if  you  have  won  the  contest  based  on  your  scores.  Thank  you. 

[Import  the  multitask  and  gauge-monitoring  task  performance  measures  into  Excel. 
Grade  all  SAGAT  responses  and  import  data.  Enter  combined  score  on  contest  sheet. 
Clean  room,  and  file  subject  folder.] 
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Level  of  Automation  Definitions 


Manual  control 


No  additional  assistance  is  provided. 
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Information  acquisition 

A  scan  line  rotates  clockwise  around  the  radar  display,  and  as  it  passes  over  an  aircraft  icon, 
a  Trajectory  Projection  Aid  (TP A)  for  that  aircraft  is  presented  for  two  seconds.  The  TPA 
shows  the  aircraft  route  in  the  form  of  a  line  connecting  the  aircraft  and  the  destination  — 
either  an  airport  or  holding  fix.  The  aircraft  speed  (in  knots),  destination  airport,  and 
destination  runway  are  affixed  to  the  center  of  the  scan  line.  If  the  right  mouse  button  is 
clicked  on  the  aircraft  during  the  two  seconds  after  the  scan  line  passes  it,  the  TPA  will  be 
displayed  for  an  additional  8-seconds.  No  information  is  displayed  in  the  automation  aid 
box.  All  other  fimctions  of  the  simulation  operate  as  in  the  manual  control  mode. 


Information  analysis 

Information  pertinent  to  each  of  the  contacted  aircraft  on  the  radarscope  is  displayed  in  the 
automaton  aid  box  including  the  aircraft’s  call  sign,  destination  airport,  destination  runway, 
speed,  and  distance  (ran)  fi'om  the  destination  airport.  A  final  block,  ‘Conf,’  denotes  the  call 
sign  of  any  aircraft  that  are  currently  in  conflict.  All  other  functions  of  the  simulation 
operate  as  in  the  manual  control  mode. 


Decision  making 

In  addition  to  conflict  alerting,  recommendations  for  conflict  resolution  are  provided  in  the 
automation  aid  box.  The  identities  of  the  conflicting  aircraft,  the  recommended  change 
(speed,  hold,  resumption,  or  airport),  and  the  aircraft  to  which  the  clearance  should  be  issued 
are  all  displayed  together.  Up  to  three  recommendations  are  displayed  in  the  automation  aid 
box  at  a  time  and  they  are  listed  in  order  of  priority.  Runway  changes  are  not  prescribed.  All 
other  functions  of  the  simulation  operate  as  in  the  manual  control  mode. 
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Action  implementation 

This  level  of  automation  simulates  the  hand-off  of  aircraft  control  from  approach  control  to 
local  tower  control.  The  tower  automatically  maintains  full  control  and  responsibility  of  all 
aircraft  within  20  nm  of  the  center  of  the  radarscope.  Automation  within  this  area  prevents 
any  conflicts  after  the  hand-off  to  tower  control  occurs.  The  action  implementation 
automation  aid  box  displays  a  tally  of  aircraft  on  the  display  by  type.  All  other  functions  of 
the  simulation  operate  as  in  the  manual  control  mode. 


